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Introduction 


Background 


With the intent of improving chronically low-performing elementary schools across the state, 
the North Carolina General Assembly (NCGA) enacted legislation in 2016 to establish a new 
non-geographic school district—the Achievement School District. In 2017, the NCGA provided 
additional guidance for the district and changed its designation to the North Carolina Innovative 
School District (the ISD). 1 Operating within the North Carolina Department of Public Instruction 
(NCDPI) as a separate North Carolina school district, the ISD is managed by a superintendent 
who works directly with the State Superintendent and the State Board of Education. Figure 1 
provides a visual illustration of the operational structure of the ISD. 

Figure 1. Operational Structure of the ISD 
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Vision and Mission of the ISD 

The ISD's charge is to work with identified schools and their communities to foster accountable, 
data-driven partnerships designed to promote and implement a shared vision of equity and 
opportunity for students in those schools. 

The vision of the ISD is to be a bold, unapologetic leader in transforming low-performing 
schools in North Carolina. Its intent is to establish its leadership role through the creation of 
strong community partnerships, strategic coalitions, and the innovative implementation of 
data-informed practices. 


1 Evaluation-relevant components of the legislation and links to the full enacting legislation are included in 

Appendix A. 
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The mission of the ISD is to improve student achievement by creating innovative conditions in 
partnership with communities across North Carolina, with a focus on equity and opportunity in 
low-performing schools. The ISD intends to accomplish this by: 

• Re-defining (expanding the definition of) what a "good" school is; 

• Aligning local and state expectations for schools—honoring community needs for their 
schools while also acknowledging state needs for the school; 

• Creating conditions for innovation (school-, community-, and policy-level) that will 
facilitate NC to broaden its perspective on how to transform all lower-performing 
schools; 

• Raising awareness and changing expectations at the school/community level (among 
parents, community representatives, and other local stakeholders); 

• Creating a sense of urgency and accountability at both the state and local levels, which 
will help re-prioritize the state's approach to education; 

• Empowering local-level stakeholders to act on that urgency; and 

• Conducting the work with a Research-Practitioner Partnership mindset. 

To meet this vision and mission, ISD leadership has designed two distinct management 
strategies for school improvement: third-party management and local-level 1-Zone 
management. 

Third-Party Management 

The first strategy involves transferring low-performing schools from their local education 
agencies (LEAs; North Carolina's formal term for traditional local school districts) into the ISD, 
where they are operated by an Innovative School Operator (ISO). In 2017, the ISD 
superintendent identified 48 elementary schools across 21 LEAs that qualified for consideration 
for the first cohort of ISD schools. In 2018, ISD identified 14 schools (eight of which also had 
been identified the previous year) across nine LEAs. In 2019, ISD identified 12 schools (five of 
which also had been identified in one or more earlier years) across 10 LEAs (Appendix B). 
Identification was based on four criteria: 

• Include all or part of grades l<-5; 

• Earned an overall school performance score in the lowest five percent (5%) of all schools 
in the state in the prior school year; 

• Did not exceed expected growth in at least one of the prior three school years and did 
not meet expected growth in at least one of the prior three school years; and 

• Did not adopt one of the already-established reform models available via state statute 
in the previous school year. 

By general statute, ISOs are described as "entities" and fall into two broad categories: 
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1. The entity has a record of results in improving performance of persistently low- 
performing schools or improving performance of a substantial number of persistently 
low-performing students within a school or schools operated by the entity in this State 
or other states. 

2. The entity has a credible and specific plan for dramatically improving student 
achievement in a low-performing school and provides evidence that the entity, or a 
contractual affiliate of such an entity, is either currently operating a school or schools in 
this State that provide students a sound, basic education or demonstrating consistent 
and substantial growth toward providing students a sound, basic education in the prior 
three school years. 

Examples of eligible entities include: 

• Established local, state, or national non-profit with a proven school turnaround record 

• College or university that employs a proven turnaround school leader(s) 

• Proven/credible charter management/education management organization 

• Corporation/business with a credible plan and a proven turnaround school leader or 
leaders 

• Proven school turnaround leader who creates her or his own entity 

• In the event that temporary management is necessary due to contract termination, lack 
of a qualified ISO, or other unforeseen emergency, the ISD is authorized to act as an ISO. 

2017. After several rounds of review, Southside Ashpole Elementary school in Robeson County 
ultimately was chosen for transfer into the ISD for the 2018-19 school year. After a competitive 
bid, vetting by an external reviewer, and approval from the State Board of Education, the ISD 
engaged Achievement for All Children 2 to manage operations at the school for five years. The 
contract establishes performance metrics that define expected progress for improvement in 
student achievement. 

The ISD superintendent managed partnerships between Achievement for All Children, parents 
and families, the LEA, and other community partners. 

2018. The ISD identified Carver Heights Elementary School in Wayne County as the second ISD 
school; however, the North Carolina General Assembly overturned the identification 3 and 
Carver Heights remained a part of Wayne County Public Schools. In January 2019, the North 
Carolina State Board of Education approved "Restart" status for the school, which means that 
Wayne County Public Schools is responsible for the school's academic turnaround. As a result, 
no new schools were identified for transfer for the 2019-20 school year; however, if Carver 


2 http://aac.school/ 

3 North Carolina General Assembly Session Law 2018-145 
( https://www.ncleg.gov/Sessions/2017/Bills/Senate/PDF/S469v8.pdf ) 
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Heights does not show academic improvement over the 2019-20 and 2020-21 school years, it 
can be turned over to the ISD for the 2021-22 school year. 

2019 Forward. In September 2019, the ISD identified 12 schools for consideration for the 2020- 
21 school year, with October 15, 2019, set as a date for finalizing recommendations and 
presenting them to the State Board of Education, which will act on the recommendations by 
January 15, 2020. 

Under current legislation, the ISD must take over five schools by the start of the 2021-22 school 
year; however, pending House (H798) and Senate (S522) legislation being considered during the 
2019 session of the North Carolina General Assembly 4 could result in significant changes to the 
identification process. In particular, the proposed legislation would require schools to go 
through a three-year process from the time of first qualification before joining the ISD. During 
that three-year window, a qualifying school could be removed from ISD consideration if it 
demonstrates academic improvement. In addition, both versions expand school operator 
eligibility to include the possibility of management by the Department of Public Instruction; the 
House version also includes the possibility of management by a consultant in conjunction with 
the LEA. These and all other pending changes are outlined in greater detail in Appendix C. 

1-Zones 

The second ISD strategy involves a more comprehensive effort with a partnering LEA that is 
hosting an ISO as part of the first strategy. In this second optional scenario, the participating 
LEA can propose operating a group of its low-performing schools via a new management model 
called an Innovation Zone (1-Zone). In the 1-Zone, the local school superintendent and school 
board are granted increased flexibility by the State Board of Education, upon recommendation 
of the ISD superintendent, with approval for five years. Flexibility may include options such as 
extending the school day, altering school calendars, and instituting creative school staffing and 
compensation models. If an 1-Zone is approved by the State Board of Education, the ISD 
superintendent provides up to $150,000 in matching funds (matched 1:1 by the LEA) for up to 
five years to support the effort. The schools in an approved 1-Zone are led by an Executive 
Director and her or his team. While the Executive Director's appointment requires the approval 
of the ISD superintendent and the State Board of Education, all other governance decisions 
remain with the LEA. 1-Zone school goals are to exceed expected growth by the last two years of 
their five-year contract. The ISD superintendent can transfer management of any 1-Zone schools 
that do not meet expected benchmarks during the last two years to the approved ISO partner 
that already is working in the LEA as part of the first strategy. 

Robeson County Schools declined the option to establish an 1-Zone when Southside Ashpole 
was selected. 


4 North Carolina General Assembly House Bill 798 ( https://www.ncleg.gov/BillLookUp/2019/H798 ); North Carolina 
General Assembly Senate Bill 522 ( https://www.ncleg.gov/BillLookUp/2019/S522 ) 
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Evaluation of the ISD 

To ensure that progress in its schools is evaluated effectively, the ISD partnered with an 
independent evaluation development team—composed of staff from the Friday Institute at 
North Carolina State University and RTI International—to develop a multi-year plan for 
assessing outcomes of the initiative. The ISD partnered with the Friday Institute to conduct the 
evaluation. While it is possible that future participating LEAs will exercise their option to 
establish 1-Zones, the scope of the evaluation is limited to the first strategy—third party 
management by an ISO. 1-Zone schools will be included in a special category of comparison 
schools for the quantitative analyses, should any participating LEAs establish 1-Zones. 

The ISD is responsible for submitting an annual report to the State Board of Education and the 
General Assembly (due by November 15 each year). The annual report includes information 
related to student performance, school-level operation, and overall ISD operation and 
management, with a focus on the following measured outcomes, as identified by statute (see 
Appendix A for more details): 

• Public school student enrollment in each ISD School, including student demographics 
(Headcount); 

• Public school student admissions processes and the number of students enrolled under 
the admissions category at each partnering ISD school (Compliance Monitoring); 

• Student achievement data, including school performance grades and student 
achievement scores and student growth scores, at each ISD school (Longitudinal 
Academic Monitoring [Raw Values over Time]); 

• Student academic progress in each ISD school as measured against the previous school 
year and against other schools located in the local school administrative unit and 
statewide (Quantitative Analysis); 

• Student discipline data in each ISD school as measured against the previous school year 
and against other schools located in the local school administrative unit and statewide 
(Longitudinal Academic Monitoring) 

• Best practices resulting from ISD school operations (Qualitative); and 

• Other information the ISD superintendent. State Superintendent, and State Board of 
Education considers appropriate. 

The ISD annual report is based in part on an annual report developed by the evaluation team; 
the current document is the first of these reports. 

In addition to the legislatively-mandates components of the public reports, the stakeholder 
version of the report also includes evaluations of some aspects of non-test score-based student 
performance, school-level operation, and overall ISD management and operation, as outlined 
in a logic model constructed with the ISD leadership team (Appendix D). 
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Evaluation Questions, Measurable Outcomes, and Analyses 


Evaluation Questions 

The enacting legislation outlines several key outcome measures for the evaluation (Appendix 
A). The legislation charges the selected ISO to: set clear goals related to higher academic 
outcomes for students; create safe and positive learning environments for children; ensure 
parent and community engagement; efficiently and effectively use taxpayer dollars; empower 
and equip teachers and school leaders to meet the goals; and hold ISD teachers and school 
leaders accountable to meet those stated goals. The ISO is directed to enter into an agreement 
with each school principal regarding specific goals for each school. 

The evaluation development team worked with the ISD superintendent's team to construct an 
evaluation strategy that retained all of the required ISO goals and also integrated additional 
goals to enable evaluators to arrive at a richer and more complete understanding of the 
outcomes of the initiative. Through multiple meetings with the ISD superintendent's team, the 
evaluation development team identified specific program goals and outcomes. Through an 
overall mission and vision statement for the ISD initiative, the superintendent's team expanded 
the ISD's legislatively-required commitments, and the evaluation development team 
incorporated that vision into a logic model for the overall initiative (a summary is included in 
Appendix D) that graphically represents how ISOs and schools will fulfill those commitments. 
The ISD superintendent's team identified short- and longer-term outcomes for ISD schools in 
the following areas: student academic outcomes, learning environments, parent and 
community engagement, school culture, leadership, and academic and fiscal accountability. 5 

Using the overall ISD logic model, the evaluation development team constructed a set of 
questions to guide the evaluation: 

Ql: Does the ISD improve student- and school-level academic growth and achievement? 

Q2. Does the ISD improve learning conditions, including changes in student behavior? 

Q3. Does the ISD contribute to changes in school-community engagement? 

Q4. Does the ISD contribute to changes in the culture of schooling both in and outside of the 
ISD? 

Q4o. Does the ISD contribute to an overall change in the culture of schooling in ISD 
schools? 


5 The evaluation team conducted a similar process to construct a school-level logic model with the first ISO and 
plans to repeat the process both with other ISOs as they are identified and annually with the first ISO as its 
implementation at the first ISD school evolves. 
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Q4b. Does the ISD change approaches to ensuring sustainable, high-quality staffing in 
ISD schools? 

Q4c. Does the ISD change approaches to leadership in ISD schools? 

Q4d. Does the ISD contribute to changes in the culture of schooling in partnering LEAs, 
non-participating LEAs, and/or across the state overall? 6 

The evaluation plan developed to address these questions meets the legislative requirements, 
while also providing greater breadth and depth of information in areas of importance to ISD 
leadership. 

Measurable Outcomes 

The evaluation development team next identified measurable outcomes that align to the 
overall goals and commitments of the initiative. Measurable outcome statements are listed 
below by evaluation question. Outcome statements in red address evaluation components 
highlighted in the enacting legislation, and statements in black provide important additional 
context for all measured outcomes. Statements in grey italics address evaluation components 
identified by ISD leadership as areas of interest for their own internal evaluation efforts; due to 
budget constraints, only some of these can be addressed in each annual report. 

Ql: Does the ISD improve student- and school-level academic growth and achievement? 

• ISD schools achieve and maintain performance scores at or above a C. 

• ISD schools' academic outcomes compare favorably to eligible but non-identified 
schools. 

• ISD students demonstrate academic proficiency. 

• ISD schools demonstrate academic growth. 

• ISD students exhibit more growth than students at matched a) local schools (possibly 
including 1-Zone schools), b) transformational schools, and c) considered but non- 
selected schools; also vs statewide growth averages. 

Q2. Does the ISD improve learning conditions 7 , including changes in student behavior? 

• ISD student support services meet statutory requirements. [Not addressed by this 
evaluation] 

• ISD schools establish a safe and positive learning environment. 

• ISD schools experience reduction of behavioral referrals. 


6 Evaluation sub-question 4d is for internal ISD evaluation planning only; questions and data collected related to 
this question are not part of the formal, public evaluation. See note below re: red and grey italics text. 

7 The evaluator should work with ISD leadership to delineate specific learning conditions to be tracked. 
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• ISD schools experience reduction of student absenteeism and/or increased attendance. 

Q3. Does the ISD contribute to changes in school-community engagement? 

• Community engagement grows in ways that are attributable to the ISD. 

o Formal relationships are established between ISD schools and community 
representatives. [Not addressed by this evaluation] 

o In-kind supports and services provided to ISD schools change in positive ways. 

[Not addressed by this evaluation] 


• Parent engagement grows in ways that are attributable to the ISD. 

o ISD parents/guardians understand academic expectations and standards. 

o ISD parents/guardians understand connections between student engagement 
and academic outcomes. 

o ISD parents/guardians participate more frequently in the life of the school, 
o ISD parents/guardians participate in the academic life of their own students. 

• ISD school staff and community member feelings of empowerment grow in ways that 
are attributable to ISD school-community engagement efforts. 

Q4. Does the ISD contribute to changes in the culture of schooling both in and outside of the 
ISD? 

Q4a. Does the ISD contribute to an overall change in the culture of schooling in ISD 
schools? 

• ISOs lead diffusion of best practices across ISD schools. [Not addressed by this 
evaluation] 

Q4b. Does the ISD change approaches to ensuring sustainable, high-quality staffing in 
ISD schools? 

• The ISD implements process to build a local, sustainable teacher workforce 
pipeline. [Not addressed by this evaluation] 

• ISD schools recruit effective staff. 

• ISD schools retain effective staff. 

• ISD schools create an effective structure for holding staff accountable. 

• ISD schools promote a culture of professional learning among teachers. 

Q4c. Does the ISD change approaches to leadership in ISD schools? 

• ISD schools exhibit fiscal efficiency and effectiveness (included in formal 
evaluation reports when provided to Team by ISD leadership). 
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• ISD schools are led by effective school administrators. 

• ISD schools adopt effective leadership strategies. 

• ISD schools establish a diffused leadership model that includes teacher 
leadership. 

Q4d. Does the ISD contribute to changes in the culture of schooling in partnering LEAs, 
non-participating LEAs, and/or across the State overall? 

• ISD leadership encourages diffusion and replication of strategies identified as 
effective practices to partnering LEAs. [Not addressed by the evaluation] 

• Non-participating LEAs demonstrate leadership-level changes in approaches to 
school leadership. [Not addressed by the evaluation] 

Data Collection and Analysis 

Thoroughly addressing these evaluation questions and measuring these outcomes requires a 
mixed-methods approach that incorporates quantitative and qualitative data from a variety of 
sources. While some of the questions focus on the performance of students in ISD schools, 
others examine the ways in which these schools operate and the extent to which they engage 
their local communities. Table 1 provides a high-level crosswalk between the evaluation 
questions and the relevant sources of data. 


Table 1. Data Sources Matched with Evaluation Questions 


Source of Data 

Related Evaluation Questions 

North Carolina administrative and Report Card data 

1, 2, 4b, 4c 

Student surveys 

2 

Parent surveys 

2, 3, 4c 

Practitioner surveys 

3, 4a, 4b, 4c 

Practitioner focus groups 

3, 4b, 4c 

North Carolina Teacher Working Conditions Survey 

2 


Appendix E provides a complete crosswalk between evaluation questions, measurable 
outcomes, indicators, and data sources; the previous evaluation report 8 includes additional 
details about the data collected from each source and connections between those data and the 
evaluation questions. 


8 Stallings, D. T., Rosof, L., Halstead, E., Knapp, L., and Rice, O. (2019). North Carolina Innovative School District 
Evaluation Commitments. Prepared for the North Carolina Innovative School District, (submitted to the North 
Carolina State Board of Education, January 9, 2019; not posted online) 
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Due to contract finalization timelines, the evaluation team was not able to commence data 
collection for this report until spring 2019. 

Because only one school opened in the first year, the evaluation plan was constructed on the 
premise that the first full baseline year would be the 2019-20 school year (the year in which 
multiple ISD schools were scheduled to open); however, as noted above, the identification of a 
second school (to be brought into the ISD for the 2019-20 school year) was overturned. As a 
result, the original intent to classify 2019-20 as the baseline year for ISD has been modified; 
now, each school's first year (regardless of start year) is classified as a "baseline year," with the 
second year of operation designated as the "growth year," and the third year of operation 
designated as the "measurable trend" year. In future reports, results from the first identified 
school will be reported separately from results in all other ISD schools, on the premise that all 
subsequent ISD schools will learn early implementation strategies from the first school's 
baseline year; i.e., implementation for all other ISD schools should reflect the final vision for the 
ISD initiative more accurately. 

Analyses of North Carolina Administrative and Report Card Data 

The full set of administrative data typically only become available to evaluators several months 
after the end of each school year (usually no earlier than December); therefore, full analysis 
and reporting of results from administrative data typically occur in spring of the following 
school year. Though school-level Report Card data were available for 2018-19 earlier than usual 
(September), they were not available early enough to allow for rigorous analysis. In addition, 
since only one ISD school opened in the inaugural year (2018-19), the evaluation team 
considers administrative data analyses included in this report to be preliminary and formative 
only. As a result, the evaluation team incorporated only descriptive analyses of 2018-19 data 
in this first report. Table 2 shows an estimated timeline for administrative data analysis for the 
four-year evaluation; Appendix F includes details about the analyses that will be conducted as 
more data become available and as more schools are added to the ISD. 

Table 2. Timing of Analysis of Administrative Data 



2018-19 

2019-20 

2020-21 

2021-22 

2022-23 

Inaugural School (opened 2018-19) 

— 

Co 

Co-i 

Co -2 

Co-3 

Additional ISD Schools (open 2020-21) 

— 

— 

— 

Co 

Co-i 

Comparison School Analysis 

— 

V 

V 

V 

V 


Key: C=Collect and Analyze; Subscript #=Student Cohort (0=Baseline, l=Growth, 2 and 3=Measurable Trend) 

Note: Data collection is delayed by one year. Anticipated completion of analysis is spring of indicated year. 

The first ISD school is an elementary school with grades pre-Kindergarten through grade 5. For 
students between grades 3 and 5, the evaluation team collects end-of-grade examination data. 
For grades K through 2, in order to capture early reading growth, the evaluation team originally 
proposed to collect mCLASS reading diagnostic examination data; however, due to changes in 
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the diagnostic tool used, 9 the evaluator will revisit with ISD officials the original strategy for 
analyzing changes in reading outcomes for students in those grades. It is important to note that 
these early-grade formative instruments are not designed for summative assessment, so any 
results presented in future reports will be reported with all appropriate caveats. All reporting is 
at the grade or school level (no individual student-level data are being collected) and includes 
Education Value-Added Assessment System (EVAAS) scores and School Report Card grades. 

Comparison Schools 

Because the North Carolina administrative data include all students in the state's public school 
system, the evaluation team had several different options for defining comparison groups. The 
team opted to compare the first ISD school not only to results from other elementary schools in 
the ISD school's original LEA (Robeson County Public Schools), but also to two statistically- 
identified comparison schools with student, teacher, and school characteristics very similar to 
those of the first ISD school (Table FI, Appendix F; see Appendix G for more information about 
the matching process). For one evaluation question (Question 1), there is a legislative 
requirement to compare the average growth of each ISD school with the average growth of all 
other qualifying (but not selected) schools; as result, the group of schools designated as 
qualifying but not selected serve as another comparison school pool (Appendix B). 

Surveys 

To compensate for the limitations in administrative data, the evaluation has been strengthened 
by the inclusion of multiple qualitative data sources, the most efficient and dynamic of which 
are the survey instruments developed for multiple audiences (Appendix H). The evaluator 
developed a common pool of survey items from which were created student, practitioner, and 
parent surveys. While each survey audience was not able to respond to each item, 
development of a common pool of questions enhanced opportunities for cross-survey analyses 
for items included in two or more survey instruments. As noted above, data collection for the 
2018-19 school year was limited to spring only; the evaluation team plans to conduct each 
survey twice a year starting with the 2019-20 school year. Full results are located in Appendix I. 

Student Surveys. The evaluator developed a survey instrument for students in grades 3 through 
5 to capture changes in student perceptions of school safety (physical, social, and emotional) 
and the overall school learning environment. 

Parent Surveys. To determine the extent to which parents understand the ISD concept and are 
satisfied with their families' experiences, the evaluator developed a survey instrument for 
administration to each ISD school's parent/guardian population each spring. The survey focuses 
on parents' perspectives on learning conditions, including changes in their own students' 
behavior, awareness of perceived changes in school-community engagement, and changes in 
school leadership. Questions about school safety capture information about parents' 


9 In July 2019, NCDPI contracted with iStation to provide reading diagnostic services beginning in January 2020. 
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perceptions of the physical as well as social and emotional well-being of their children while at 
school. Questions about a positive learning environment probe for responsiveness of leadership 
to parents' concerns. Questions about school-community engagement reveal parent knowledge 
about academic expectations and standards, opportunities for student engagement, level of 
parent engagement, and perceptions of school leadership. 

Practitioner Surveys. The evaluator also developed a practitioner survey instrument to be 
administered to administrative staff and teachers at each ISD school during the spring of each 
academic year. The practitioner survey focuses on ISD contributions to school-community 
engagement, as well as ISD approaches to sustainable, high-quality staffing and school 
leadership. Questions about school culture and school-community engagement capture 
information about staff perceptions of ISO-provided support for implementation of best 
practices and for dealing with parents and the community. The survey also gathers information 
about staff perceptions of accountability, culture of professional learning, and both school and 
teacher leadership. 

North Carolina Teacher Working Conditions Survey. The North Carolina Teacher Working 
Conditions survey 10 is administered statewide in the spring of every even-numbered year by 
NCDPI and supplements the data collected in the recommended practitioner survey with 
additional data that are comparable over time with responses from other schools. Because no 
survey was administered during the 2018-19 school year, this data source will be used for the 
first time in analyses to be completed for the second report (fall 2020), pending timely 
availability of results. 

Focus Groups 

To supplement practitioner survey data, the evaluation team conducted focus groups of a 
representative sample of the practitioners who are involved in ISD school operations in spring 
2019 and will continue to conduct such focus groups at the end of each academic year. 
Practitioners included classroom educators, school staff, and school and state-level 
administrators. These focus groups allowed the evaluation team to address more deeply 
questions about school culture, leadership, accountability, empowerment, parental and 
community engagement, and the learning environment created through ISD schools (and how 
those environments serve students' needs). 


10 https://ncteachingconditions.org/ 
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Results from the First Year of Implementation 


As with any major initiatives undertaken by the state or its agents, there is an important 
distinction between the evaluation of the ISD initiative and the outcomes of that initiative in a 
single school setting. The ISD in 2018-19 was represented by a single school, but a fully-realized 
ISD would include up to four other schools. Measurement of the impact of the initiative would 
be based on outcomes at all of those schools, with an expectation that performance across 
those five schools would reflect variability in the levels of success experienced at each. 

Consequently, since only one school constituted the ISD during its first year, the evaluation 
team strongly cautions against evaluating the initiative as a whole based on single-year 
outcomes from a single school. As noted in the previous section, the evaluation team considers 
the 2018-19 implementation year to be a pilot year, with the ISD, its original Operator and 
school, and future Operators and schools all able to (and expected to) benefit from the lessons 
learned during this start-up year. 

In addition, while the evaluation team understands that it has become common practice in 
North Carolina for most examinations of school quality to focus on a review of test scores (and 
this report does the same by leading with the limited student outcome data available at the 
time of its completion), we believe this report would be incomplete if it ended its examination 
with the review of test scores. Instead, we have used data from all of the sources described 
above to provide what we hope is a deeper, more qualitative assessment of some of the 
possible reasons behind the Year 1 academic outcomes. While we cannot draw a causal line 
between the Year 1 academic outcomes and the qualitative findings, we believe those findings 
may help provide a richer context for understanding some of the possible reasons behind the 
numbers. These findings also may reveal some avenues for strengthening ISD implementation 
in subsequent years. 

Academic Growth and Achievement 


Ql: Does the ISD improve student- and school-level academic growth and achievement? 


Preliminary Academic Outcomes 

Overall Outcomes. As detailed in an earlier section, most academic results for the 2018-19 
school year were not available early enough to allow for rigorous analysis before this report 
was finalized; however initial School Report Card data were released in September 2019, 
allowing us to include some data related to academic outcomes. School-level outcomes for 
Southside Ashpole, the two matched comparison schools, and the five other finalist schools 
from the 2017 selection list are included below (Table 3 and Figures 2 and 3, following pages); 
school-level results for the larger list of comparison schools (other schools on the 2017 ISD list, 
other Robeson County elementary schools) are included in Table FI (Appendix F). 
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As demonstrated in the table and figures, overall, academic performance for Southside Ashpole 
at the end of the 2018-19 school year on several different measures appears to have changed 
little from performance over the four preceding years. Academic performance for the two 
statistically matched comparison schools (schools statistically similar to Southside Ashpole but 
not subject to ISD intervention) followed a similar, relatively flat trend, but it is worth noting 
that most of the five other finalists from the 2017 ISD pool experienced increases in student 
academic performance in one or more measures, with four of those schools moving from an 
overall 'F' grade in the year prior to the selection year (2016-17) to either a 'D' or 'C in 2018-19. 


Table 3. School Report Card Grades and Scores, 2014-15 through 2018-19 



School Report Card Grade/Score 

School Year 

2014-15 

2015-16 

2016-17 

2017-18 

2018-19 


ISD Schools 


Southside Ashpole 

F 

36 

F 

35 

F 

27 

F 

29 

F 

30 

Statistically-Matched Elementary Schools 

Matched School A 

F 

29 

F 

33 

F 

33 

F 

39 

F 

36 

Matched School B 

F 

36 

D 

40 

F 

29 

F 

25 

F 

26 


ISD Finalist Schools 


Glenn Elementary 

F 

39 

F 

36 

F 

37 

D 

50 

D 

42 

Lakewood Elementary 

F 

38 

F 

35 

F 

35 

F 

37 

C 

55 

Williford Elementary 

D 

41 

F 

37 

F 

26 

F 

30 

F 

34 

Willis Hare Elementary 

D 

49 

D 

48 

F 

36 

D 

51 

D 

49 

R B Dean Elementary 

F 

34 

F 

36 

F 

36 

D 

40 

D 

43 


Figure 2. School Performance Grade Scores, 2014-15 through 2018-19 
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Figure 3. Performance Composite , 2014-15 through 2018-19 
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Disaggregated Outcomes. Equally as important as overall results are results for sub-groups, as 
school-level results sometimes can hide progress for smaller clusters of students or for 
individual subject areas. Subject- and grade-level outcomes were presented to the State Board 
of Education in September 2019 11 but were not available in time for this report to include 
analyses alongside results from comparison schools. Presented here are the 2017-18 and 2018- 
19 results as shared with the State Board. On the surface, there appear to have been gains in 
mathematics between 2017-18 and 2018-19 alongside losses in reading (Table 4). 

Table 4. Changes in Growth , 2017-18 to 2018-19 


2017-18 

2018-19 

Overall 

Not Met 

Not Met 

Reading 

Met 

Not Met 

Mathematics 

Not Met 

Met 



That interpretation is tempered somewhat when we examine grade-level scores. One of the 
challenges associated with comparing grade-level scores over time is that the students in those 
grades change year to year. Thus, an "improvement" between 2017-18 and 2018-19 in (for 
example) Grade 5 mathematics scores may have as much (or more) to do with differences in 


11 https://simbli.eboardsolutions.com/Meetings/Attachment.aspx?S=10399&AID=188144&MID=5735 
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the ability levels of the incoming new Grade 5 students and the Grade 5 students from the 
previous year as it has to do with improvements in instruction in Grade 5. Thus, many of the 
outcomes in Table 5 are presented so that scores for a given grade in 2018-19 are placed beside 
scores for the previous grade in 2017-18—the grade in which most if not all of the 2018-19 
students would have tested in 2017-18. 12 

Table 5. Changes in Grade-Level Proficiency and College- and Career Readiness Rates, 2017-18 
to 2018-19 



Grade-Level Proficient 

College- and Career-Ready 


2017-2018 

2018-2019 

2017-2018 

2018-2019 

All Subject Areas 

Grades 3-5 

20.2 

19.8 

11.3 

6.6 

Grade 2 to 3 

32.1 


10.3 

Grade 3 to 4 

22.6 

12.5 

11.3 

5.6 

Grade 4 to 5 

20.2 

15.7 

13.1 

* 

Grade 5 to 6 

18.4 


7.1 


Reading 

Grades 3-5 

26.2 

19.8 

13.2 

9 

Grade 2 to 3 

20.5 


7.7 

Grade 3 to 4 

28.6 

22.2 

11.3 

11.1 

Grade 4 to 5 

33.3 

16.7 

16.7 

8.3 

Grade 5 to 6 

22.2 


8.3 


Mathematics 

Grades 3-5 

12.3 

21.6 

7.9 

5.4 

Grade 2 to 3 

43.6 


12.8 

Grade 3 to 4 

16.7 

* 

8.2 

* 

Grade 4 to 5 

7.1 

16.7 

9.5 

* 

Grade 5 to 6 

13.2 


* 


Science 

Grade 5 

26.3 

13.9 

8.3 

* 


*=5 or fewer students 


12 Some readers may ask whether these two groups of students are comparable; i.e., whether the student 
population at Southside Ashpole changed significantly—either overall or for certain grades—between 2017-18 and 
2018-19. In general, the population was relatively stable, with about 88% of Grade 3 through Grade 5 students 
returning from the previous year (from Grade 2 through Grade 4). However, taken separately, there was much 
greater turnover for Grade 3, with only 78% returning from Grade 2 in 2017-18 (compared to 95% returning in 
Grades 4 and 5), raising a question for future analysis about differences in proficiency between returning Southside 
Ashpole students and students new to Southside Ashpole. 
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When examined with this year shift in place, comparing results for the same students across 
two years, many of the apparent gains in mathematics are less compelling than they may 
appear if scores are compared for the same grade level across two years. For example, while 
Grade 3 mathematics proficiency rates moved from 16.7% proficient in 2017-18 to 43.6% 
proficient in 2018-19, the two groups of students are not the same. The biggest boost in 
mathematics when we compare results for the same students across two years appears to be 
for students who were in Grade 4 in 2017-18 (only 7.1% proficient) and those same students in 
Grade 5 in 2018-19 (16.7% proficient). Indeed, much of the overall boost in mathematics 
proficiency from 2017-18 to 2018-19 (from 12.3% to 21.6%) is explained primarily by the 
relatively high proficiency rate of the Grade 3 students in 2018-19—more than 20% of whom 
were not enrolled at Southside Ashpole in 2017-18 (see footnote 12). 

Still, mathematics scores were higher in 2018-19 than they were in 2017-18. Determining the 
extent to which those higher proficiency rates in 2018-19 are attributable to improvements in 
instruction and learning relative to the extent to which they are attributable to differences in 
grade-level populations will require a deeper examination of the Grade 2 2017-18 mathematics 
proficiency levels of the 2018-19 Grade 3 students, as well as a close watch for the emergence 
of any multiple-year trends. Indeed, as we explore in the following sections, several teachers 
noted both that the new curriculum seemed to provide greater structure and that the younger 
students at Southside Ashpole seemed to respond more positively to that curriculum. 

Curriculum 

We begin our qualitative investigation of some of the possible factors behind the Year 1 
academic outcomes with a brief review of the adoption and implementation of the school's 
new curricula. 

Before the start of the school year, Southside Ashpole implemented new mathematics and 
Language Arts curricula. The curricula—Common Core Language Arts (CKLA) and Eureka Math — 
were chosen by XXX because XXX. Since none of the teachers at Southside Ashpole had used 
either curriculum before, the school and Operator had planned to provide professional 
development throughout the year to support implementation of both curricula; however, some 
teachers indicated that professional development was not sufficient (see Changes in 
Approaches to Ensuring Sustainable , High-Quality Staffing, below). As one teacher observed, 
"Professional growth for me [this year wa]s just learning this curriculum." 

While most teachers indicated in focus groups that they liked the curricula and said both 
provided students with the structure and content necessary to improve their knowledge and 
skills, some thought they were too rigorous, based on student academic readiness at the start 
of the school year: "It's so beyond what they have been exposed to that it's so hard." The 
teachers did believe that, with time, students would become more comfortable with the 
curriculum and that they would be able to see gains in student achievement. Teachers also 
indicated that there was no curriculum for science (where Grade 5 proficiency rates were 
halved between 2017-18 and 2018-19; Table 5), social studies, or for the specials. The specials 
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teachers indicated that they tried to tie their lessons to what students were learning in class 
and decided to use the North Carolina standards to guide their curricula. 

Teachers also realized as the year progressed that certain North Carolina Standards were not 
covered by the new curricula; as a result, they had to adjust lessons to include these missing 
Standards: 

[Ojnce we realized that, then we've started adding in those pieces that they were missing. But 
obviously with the hurricane and everything that's going on, we're not going to get to cover 
everything, but we're going to do the best that we can to cover the missing pieces because it 
wasn't until just six weeks ago that we even discovered there were missing standards. 

All teachers agreed that they try to use student data to improve instruction. Southside Ashpole 
used MAP® Testing (Measures of Academic Progress®) to benchmark student progress, but 
some teachers believed that the tests did not always align with the curricula, which made some 
of the data less useful. Among administrators, only the principal was able to comment directly 
on the use of student data to improve instruction. He noted that, while the school explored 
ways for data to inform instruction (indeed, on our first site visit, we saw evidence of such 
conversations in the staff planning room), staff use of data "isn't at the level I'm accustomed 
to"—something he attributed to general lack of experience in that area across all staff. The 
school did provide professional development in data use, but that professional development 
was limited to within-school trainings, with no opportunity for any staff to gain a broader 
perspective from external professional trainings. The disconnects between chosen curricula and 
state standards is a significant challenge, and one that echoes loudly in the larger conversation 
about the degree to which each ISD school is able to operate with autonomy (see Broader 
Observations, below). 

Learning Conditions 


Q2. Does the ISD improve learning conditions, including changes in student behavior? 


Student Behavior 

The Operator and administration at Southside Ashpole introduced new student behavior 
policies and procedures as part of the operational changes at the school. Focus group 
participants indicated, however, that some older students—as well as some teachers— 
struggled to adjust to the changes, primarily because the new policies were different from 
those to which they were accustomed. 

In addition, some teachers indicated that, because their philosophy of discipline did not align 
with the Principal's, they sometimes chose to handle discipline independently. This discrepancy 
is significant and contributed to some of the tensions between staff and administration that are 
explored in greater detail in the School Culture section, below. An upper-grade teacher shared 
this perception of the philosophical divide: "I would never write a referral anymore on my kids. 
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... I'm going to call their parents because I know something's going to happen then.... If I 
send them to the office, they're just going to be chilling in the office." 

Despite difference of opinion between administration and some teachers with respect to how 
to handle behavior problems, many staff believed behavior and engagement improved over 
time as teachers and administration showed that they cared about the students. 

Student Attendance 

Almost all surveyed parents (n=17 households, 23 students) agreed that regular school 
attendance is absolutely essential or very important to their children's success in school. At the 
time of this report, all data for the 2018-19 school year were not available yet, but trend data 
from previous years will make it possible in the future to identify any significant changes in 
student attendance rates. In the years leading up to its inclusion in the ISD, Southside Ashpole's 
month-to-month student absenteeism rates were comparable to those of the statistically 
matched comparison schools (Figure 4). 

Figure 4. Pre-ISD (Historical) Student Absentee Rates, by Month, Southside Ashpole and 
Comparison Schools, 2014-15 through 2017-18 
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Teacher Attendance 

Data on historical and current teacher absenteeism rates were not available in time for 
inclusion in this report. 13 They will be included in future reports when possible. 


13 In a September 2019 presentation to the State Board of Education 

( https://simbli.eboardsolutions.com/Meetings/Attachment.aspx?S=10399&AID=188144&MID=5735 ), the ISD 

provided some preliminary teacher attendance data for the start of the 2019-20 school year. 
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School-Community Engagement 


Q3. Does the ISD contribute to changes in school-community engagement? 


Both the ISD and Southside Ashpole made changes to how the school engaged with parents and 
the community. In previous years, many events were held during the day, which was not 
convenient for many parents. For 2018-19, the school offered more evening events to allow 
more parents to participate. For example, one teacher observed that, for an awards event, 
"everybody came out." The school also hosted a data night, a Fall Festival, award ceremonies, 
and a concert. A teacher who assisted with the concert was told parents would not come, but 
the teacher happily reported: 

The gym was packed, we ran out of seats. It really exceeded my expectations and just 
the community coming out. I got to talk with parents I have not seen[; there were] 
children that have come up and said, "This is my mom. This is my dad," and the parents 
will say, "Thank you for having this so late because we get off of work at 5:00." 

A high percentage of parents also reported on the Parent Survey (n=17 households, 23 
students) that they felt welcomed at the school (82%), felt comfortable talking with 
administrators (71%), and felt comfortable talking to teachers (76%). Flowever, only about half 
of all parents (53%) indicated that the school provided them with clear information about what 
their children were learning in school. 

School Culture 


Q4. Does the ISD contribute to changes in the culture of schooling both in and outside of the 
ISD? 

Q4o. Does the ISD contribute to an overall change in the culture of schooling in ISD 
schools? 

Q4b. Does the ISD change approaches to ensuring sustainable, high-quality staffing in 
ISD schools? 

Q4c. Does the ISD change approaches to leadership in ISD schools? 


In addition to the lack of ISD-era data for measuring longitudinal changes in the culture of the 
school, the majority of teachers and administrators did not work at the school in previous 
years, so most only had access to limited information for making before-and-after school 
culture comparisons. The notes below represent observations from administrators, teachers, 
and parents primarily from one school year, with comments about changes over time added 
when feasible. 

Overall Changes in the Culture of the School 

Teachers. In focus groups, several teachers described their perceptions of the school's culture 
the year before ISD, as well as ways in which they believed the culture had changed. With 
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respect to student culture, one teacher (who had not been there the year before) said that her 
students noticed a positive change: 

Last year they just watched movies all day. They didn't do anything. They tell me all the 
time, "[Teacher], I wish we would have learned like this last year." So they really didn't 
do anything last year. They just chilled, kicked back, [ate] popcorn and ha[d] a great 
time .... So now it's completely opposite, they say. 

Opinions among teachers differed with respect to whether parents' interactions with students 
and teachers had changed. One teacher characterized the school as a "typical" low-performing 
school "with very little positive parent interaction," and that this culture was "the way it was 
last year." One teacher who had worked at the school before ISD believed that some parents 
took advantage of the school's change in culture: 

I've been here again [multiple] years and I've never seen parents just come in, or I guess 
they're parents, I don't know. It's like they have access ... even though the doors are 
supposed to be locked. Most of the time they're not. They just walk to and from the 
buildings when they [are] ready. I've never seen that happen [here] before. 

Some teachers even shared episodes of parents who exhibited aggressive behavior (e.g., 
incidents in which a parent had to be escorted off campus by police), but one teacher noted 
that there were fewer such incidents this year: "I mean ... that has not happened here this 
year as often as they did the last year. Last year... I knew of three or four different instances 
where that happened." 

Parents and Students. In survey responses, parents and students were split regarding their 
experiences and comfort at school, but over half of responding students (n= 46) agreed that 
they felt safe at school (59%), had an adult to help them if they felt threatened at school (67%), 
and had an adult to help them if they were bullied outside of school (54%). Parents (n=17 
households, 23 students) had similar response levels to these items. 

Administrators. There was general agreement across administrators at all levels—from the 
Principal to the Operator to the ISD—that the school's overall culture changed at least twice: 
first, in positive ways, relative to the previous school year, but then in less-positive ways as the 
school year progressed. 

Administrators (many of whom were able to spend significant time at the school in the year 
before ISD conversion) saw the most positive shifts in student attitudes and behaviors—said 
one administrator, "I've never met a more eager group of students in my life"—but early 
positive changes were true for the school as a whole as well. Where before the school 
atmosphere could be chaotic and disorderly at times, with freedom of movement for students 
and generally passive teaching behaviors on the part of classroom teachers, administrators 
noted an immediate change in the sense of order present in the school, as well as in the level of 
teacher engagement. One administrator attributed some of these changes to the presence of 
uniforms but said that their impact likely was only one factor of many. Relative to other grades. 
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5 th grade posed the greatest challenge across the year, in terms of discipline. One administrator 
noted that the shift in student behavior and attitudes might have gone even farther, had the 
school adhered more closely to its plan to incorporate the Core Virtues curriculum (a daily, 20- 
to 30-minute character-building curriculum) and the 45-minute end-of-day wrap-up alongside 
the Language Arts and mathematics curricula. 

The second culture shift took place midway through the year and was characterized by multiple 
administrators as a division among faculty and staff with respect to where each turned for 
leadership—some to the principal, but others to the ISD administration. While generally 
positive and consistent at the start of the year, rifts between the principal and some faculty 
emerged as the year progressed. These rifts perhaps were due at least in part to significant 
changes in leadership at the state level and subsequent changes in the ways in which state 
leadership interacted with Ashpole leadership, faculty, and staff. According to several sources, 
ISD leadership became more directly involved in the day-to-day administration of the school. 

Changes in Approaches to Ensuring Sustainable, High-Quality Staffing 

The Operator had a limited amount of time to hire staff for the first year, and, as a result, may 
have had a smaller-than-usual pool of talent from which to recruit. In addition, teachers 
identified aspects of their job and work environment (such as availability of support staff and 
professional development opportunities) that could impact the school's ability to recruit and 
keep high-quality staff going forward. 

Teachers across grade levels indicated that standard classroom observations took place 
throughout the year, but other aspects of support for high-quality teaching were less 
consistent. For example, some teachers felt that the school did not provide enough support 
staff for teachers. Although the school employed classroom assistants for kindergarten through 
second grade, a teacher reported frequently being without her assistant: 

I've never been used to, in [my grade], not having an assistant in the room.... [a]nd this 
year that has been a challenge for me.... [A]fter December, my assistant has been 
borrowed frequently, quite a bit. Take two weeks for example, I was without support for 
61 hours, two weeks. And with [my grade, students are] needy. They have a lot of 
needs.... [a]nd I just feel like that, not having that support, it's impeded learning. And 
my children, there's been things I wish I could have done beyond the curriculum. 

A teacher in an upper grade (who was not provided an assistant) talked about her need for an 
assistant and using parent volunteers as a supplement: 

I just need some support. Because, just like [another teacher] says she needs an 
assistant, I need one, too. Because I can't be the parent, I can't be the teacher, I can't be 
the counselor, I can't be the nurse. I can't be all of these things at one time. I have 
parents come and volunteer to help me, that's how much it is [needed]. It's [been] really 
tough. 
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In addition, as noted above, some teachers believed that there were limited opportunities for 
professional development during the school year, beyond what was provided during weekly 
planning and meeting time. At least one said that teachers had to seek out and identify their 
own opportunities and get approval from the ISD; while some teachers took the initiative to do 
so, many did not. Administrators said that their intent was for teachers to be able to use at 
least two of their planning periods each week for a joint planning session with same-grade 
teachers, but that these sessions did not materialized regularly across most grades. 

Teachers also indicated that opportunities to socialize with each other is an important 
component of retaining staff, but that typically there was little socializing among teachers at 
Southside Ashpole. Some posited that the reason was a combination of the layout of the school 
and the daily schedule. One teacher said, "Working with our colleagues, you really don't see 
them at all," and another suggested that, based on previous teaching experiences, there could 
have been more opportunities: 

[W]e have, in the past, had opportunities to interact with each other through socials and 
functions and outings and trips and workshops and things that we've done as a whole, 
but during this year, you don't have time to interact... other than entering and leaving 
[the building],... [W]e had [one] staff development where we did interact with each 
other and we were in different groups with the ones that we don't work with, so that 
was good, getting to know each other. But [usually there is] no social life at all here. 

Finally, some teachers said that they felt that not only was the instructional school day (8:00 
AM to 4:00 PM) too long for students, but also that the full work day (7:30 AM to 4:30 PM) was 
too long for teachers. 

Teachers did note that one benefit of the ISD was better resources for their classrooms, 
compared to the resources provided by Southside Ashpole's previous LEA (Public Schools of 
Robeson County). According to one teacher, "Public schools of Robeson County have not gotten 
basal readers for at least twelve years. They have not purchased reading material. And teachers 
have had to go out and pull, based on the standards, and pull their material for their 
classroom." 

Changes in Approaches to Leadership 

While administrators indicated that the intent was for teachers and staff to participate in 
decision-making to a greater degree than had happened at the school in previous years, one 
administrator acknowledged that, after a good start, in the end, the year was characterized 
more by a top-down approach to leadership, with limited opportunity for teacher or parent 
involvement in school decision-making. Most teachers felt that they could approach school 
leadership with concerns, but there also was ongoing tension with respect to some 
administrative decisions. 

Within their classrooms, teachers experienced varying levels of autonomy, depending on the 
subject. As noted earlier, the school's Language Arts and mathematics curricula were chosen by 
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the Operator as part of the application process, so no teachers were able to contribute to that 
decision. In addition, teachers were expected to stick closely to curriculum timing guidelines, 
which only gave them flexibility in how the material was presented. As one teacher noted, "A 
lot of creativity is eliminated with this curriculum." On the other hand, since no specific 
curricula were provided for science, social studies, art, or music, teachers had much more 
decision-making freedom in those areas. 

In survey responses, a large majority of teachers (83%) agreed or strongly agreed that they 
were encouraged to take on leadership opportunities. Teacher focus group discussions 
suggested that the primary formal opportunity for teacher leadership was the School 
Improvement Team (SIT), which was composed of two teachers, the school administration, and 
two parents. Some teachers indicated, however, that they were not aware of the SIT; another 
teacher noted that the SIT met irregularly—"We're supposed to meet once a month, that 
doesn't always happen"—and that when it did meet, it often was little more than a venue for 
administrators to hear feedback from teachers: 

We talk about what's going on, some of the issues that are going on. Or [the principal] 
explains what's coming up and asks our opinion.... One of the questions they always 
ask is[, "l]s there anything that you are aware of as far as other teacher concerns that 
anybody has brought to you or that you're aware of?" 

Most teachers did feel that they had the opportunity to express their thoughts or concerns 
about leadership informally. On the teacher survey, 58% agreed or strongly agreed that school 
leadership asked them for their opinions, and 67% agreed or strongly agreed that, if they had a 
concern about curriculum or policy decisions, they felt comfortable talking to someone in 
school leadership. In a focus group, a teacher elaborated that the principal "gives you the 
respect to listen. To listen to you and go in and discuss.... He's always saying his door is open 
and if you want to see him, you can go in or whatever. So he does give you that opportunity." 
Parents felt similarly able to talk to school administration: In their survey responses, 71% 
agreed or strongly agreed with the statement, "I feel comfortable talking to administrators at 
my child's school." As noted in an earlier section, however, teachers and administrators alike 
indicated that there was ongoing disagreement about things like the school's approach to 
discipline, and administrators admitted that, by the end of the year, there was a clear split 
between those who looked to the principal for leadership and those who began to look to ISD 
officials for leadership instead. 

Broader Observations 

This report has repeated the caveat often that, because the first year of ISD evaluation was 
based on data collected from one school only, most of the formative findings are relevant only 
at the individual school level. Even given the limitations of a single-school sample size, however, 
the first year of evaluation work also produced findings that may be useful for ISD 
implementers, potential participating schools and districts, and potential operators more 
broadly. Though these findings are more general than those presented earlier in this report, the 
same caveats apply: Because they are based on only a single year of observations and data 
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collection, on data collected at only one school, and on data from the first year of operation, 
the degree of confidence with which we assert their relevance to ISD more broadly—whether 
now or in the future—is limited. 

Strengths and Successes 

Southside Ashpole Made Some Strides toward Supporting Future Academic Growth. It is not 

unusual in school turnaround situations for there to be some tension between allowing 
teachers to approach their curricula in ways that they believe highlight their personal 
approaches to teaching and requiring teachers to follow a prescribed approach that allows for 
less teacher input but that is more consistent across classrooms. As noted earlier, for its first 
year as an ISD school, the Language Arts and mathematics curricula at Southside Ashpole were 
characterized more by the latter approach. Teachers at several different grade levels noted 
that, while the inflexibility in the curriculum was challenging for them (for example, they found 
it difficult to differentiate, and students in higher grades who had become accustomed to a very 
different approach to curriculum struggled the most to adjust), in their opinion the curriculum 
did provide the students with the structure and content necessary not only to improve their 
knowledge and skills but also to engage them in learning. Teachers made these estimates of 
degrees of improvement and levels of engagement relative to the quality of the academics 
provided in previous years, which by most accounts was extremely low, and early quantitative 
measures of student outcomes show that test scores were no better this year than in previous 
years, but the full payoff from a radical change in curriculum likely will take more than one year 
to detect. 

Challenges 

As earlier sections of this report have detailed, there were many challenges at Southside 
Ashpole during Year 1, and those challenges grew as the year progressed. 

Successful Implementation Takes Time. For most complex education initiatives, successful 
implementation often does not occur until at least the second year, and the same may be true 
at Southside Ashpole. Much of the first year was about experimentation, learning, and survival. 
Several teachers and administrators admitted that the first year was a learning year, and that 
they believed they would not be able to implement a shared, functional plan until at least the 
beginning of the second year. That assessment now may be somewhat optimistic, as the 
majority of the first-year staff have departed and the principalship and the Operator's 
leadership have changed—the school opened in Year Two with an almost entirely new team of 
educators. 

A good example of the learning curve challenge was the general sense of disorder evident in 
the school's day-to-day operations during the evaluation team's spring visits. There did not 
appear to be any formal procedure for checking in parents, students, or guests; several 
students wandered the hallway unaccompanied; and the front office sometimes was left 
unattended (with students present). To be sure, some of this disorder could be attributed to a 
relaxed culture carried over by students and parents from previous years, but it also suggested 
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a school that had not yet been able to find its footing with respect to discipline and other 
norms. 

The start-up challenge extends beyond any single ISD school to the ISD as a whole, which for 
the entirety of the first school's first year was being developed in real time along with that first 
school. Even with access to the lessons learned from this first pilot school, any new ISD schools 
also likely will require at least a year of start-up growth, as one of the key tenets of the ISD 
model is not to replicate intervention across schools but instead to innovate relative to the 
needs of each school (even to the point of having different Operators for each school). 
Complicating matters even more is the degree to which school leadership and staff must 
integrate into a new community and earn that community's acceptance and support—both 
school-level and state-level administrators shared as much during interviews. 

Ultimately, as noted throughout this document, success should be measured not by results 
from a single year alone, but instead by sustained success across years, even as the school's 
leadership and student population both change. These longer-term outcomes will reflect more 
than just the restoration of stability to the school —raw improvement alone should not be the 
goal. The truer measure will be the improvement achieved beyond any initial bump in 
achievement attributable to improvements in school operations. 

Successful Implementation also Requires Mutual Understandings across Leadership Entities. On 

paper, the ISD leadership structure is complex, with each school led by a Principal, an Operator, 
and a state Agency, and with employees of each school hired by different combinations of 
those entities (for instance, ISD Principals report to Operators, but ISD teachers technically are 
employees of the state Agency). This complexity was palpable in Year 1, when a single school 
was the focus of all three entities. The clearest challenge resulting from this complexity was the 
nearly year-long negotiation across all three leadership entities with respect to their 
interpretations of the phrase, "charter-like flexibility." The struggle to reach collective 
agreement on the meaning of the phrase was at the root of much of the tension that developed 
among the leadership entities—a tension that grew rather than subsided as the year 
progressed. Even the parameters and procedures for simple actions (such as small-ticket 
purchases) became opportunities for conflict alongside larger decisions (such as staffing 
changes). The frustration was so great that, at one point, one leader contended, "I've never 
[experienced] this much bureaucracy." Without clear, consistent, and comprehensive 
resolution of each party's understanding of the operationalization of charter-like flexibility in 
the ISD school setting, the most debilitating problems that surfaced during Year 1 likely will 
persist, if not worsen—not only at Southside Ashpole but also at any future ISD schools. 

There is Danger of Evaluation Misinterpretation. As noted above, ISD is an initiative, but 
Southside Ashpole is only a single school that (to date) has presented a single response in a 
single setting in a single year to the possibilities of that initiative. Whether the school succeeds 
or fails ultimately may define success or failure for the initiative as a whole, but perhaps 
unfairly so: A single school's single-year experiment should not bear the burden of representing 
a full, multi-school, multi-year implementation. 
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The challenge extends beyond results from this first year. One of the philosophical 
underpinnings of the ISD concept is that each school requires a customized plan in order to 
address each school's unique circumstances. Even the most successful plan possible for 
Southside Ashpole may not work elsewhere. Because, by design, implementation will vary from 
ISD school to ISD school, and because even at full operation there only will be a handful of 
schools, there likely never will be an opportunity to use data from multiple schools to evaluate 
rigorously any single reform approach—only an opportunity (at scale) to evaluate the presence 
of the policy. Evaluation of the success of any given implementation will be school-level only. 

A related challenge is the mismatch between the curriculum freedom given to Operators and 
the state-mandated measurements of academic growth used to determine ISD school success. 
Unless Operators are allowed to choose the academic measures that they believe best reflect 
the curriculum taught at their schools, or unless Operators are required to limit their curriculum 
choices to those that are best measured by the state's tests, there likely always will be at least 
some disconnect between actual student growth and student growth as measured by state 
tests. 
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Formative Recommendations 

Beyond the success or failure of any single school within its jurisdiction, the central question 
the ISD faces is, "What is required to establish, build, and run a school district ?" The formative 
recommendations below, derived from the findings described in earlier sections, are intended 
to help inform the state's efforts to address that question. 

Establish a Common Definition of "Charter-Like Flexibility " u 

Above all else, in order to make progress at Southside Ashpole and at any other future ISD 
school, the state and its operators should work to reduce overlaps in state-. Operator-, and 
school-level governance by reaching agreement on a common understanding of the flexibility 
allowed the Operator in running each ISD school. This understanding should be not only with 
respect to general school administration, but also specific decision-making rights and 
responsibilities. Many of the leadership and management problems encountered during Year 1 
at Southside Ashpole are not likely to be resolved until such agreement has been reached. 

A key element of this lack of agreement is each administrative level's interpretation of the 
appropriate role of the state ISD office: Is the role to provide support to each ISD school as it 
develops its own approach to managing a school, or is the role to moderate a balance between 
individual school autonomy and a core set of ISD principles? In other words, what decisions can 
be made locally, and what decisions need to be vetted at the state level? For example, after a 
hurricane closed schools throughout the region, did the decision to re-open (which in the end 
was made by the ISD) belong to the school, the Operator, or the ISD? As one administrator put 
it, "We can't have multiple drivers making multiple decisions." The principal expressed similar 
concerns: 

[Tjhere's Robeson County, ISD, the State School Board, AAC, Team CFA, and there's 
.. . me.... That's too many people involved.... I've got five [leadership levels making 
decisions], 

Fluman resources practices are another area of management that could be improved with 
greater clarity about roles and responsibilities. Most prominent of these is the reporting 
structure: The principal is an employee of the Operator, but the teachers are employees of the 


14 The most relevant statute— 115C- 75.7. Selection of innovative schools 

( https://www.ncleg.gov/EnactedLegislation/Statutes/PDF/BySection/Chapter 115C/GS 115C-75.7. pdf )— does not 
include the phrase "charter-like flexibility"; however, this phrase was the phrase used most often by each 
leadership entity (Principal, Operator, and state Agency) during focus groups and interviews conducted for this 
evaluation to define—differenlty—the Operator's parameters. Subsection (e) ("Waivers for Innovative Schools") 
states in part: "[l]nnovative schools shall be required to comply with, at a minimum, the statutory requirements for 
charter schools as provided in Article 14A of this Chapter. ... [T]he State Board of Education may grant a 
requested waiver of State laws or rules for an innovative school pursuant to this subsection, except for a waiver of 
State laws or rules applicable to children with disabilities and any of the other requirements set forth in this 
subsection." 
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ISD. One of the challenges created by this arrangement was an ongoing disagreement about 
which leadership entity (the principal, the Operator, or the ISD) makes decisions about the 
continued employment of faculty and staff—a disagreement that, at least in part, appears to 
have been responsible for some of the conflicts between school-level administration and 
staff. 15 

Also, without a clearer sense of each administrative unit's appropriate role in the management 
of the school, changes in personnel at any level can lead to changes in role interpretation—as 
was the case when there were changes in personnel for several leadership positions during the 
2018-19 school year. 

Finally, and though there is not yet enough data to state so with conviction, there was some 
indication after Year 1 that ISD schools may not require their own governance division or 
designation; most of what is expected of ISD schools possibly could be accomplished with 
support from pre-existing state school turnaround support structures. 

Align Curriculum Requirements and Measurements of Success 

The practice of letting schools choose a curriculum but then evaluating them using state 
measurements will not always allow for accurate measurement of the success of each school. If 
ISD schools are to be evaluated on their ability to meet specific state academic standards and 
objectives, then Operators should be required to propose curricula that address those 
standards and objectives; however, if the interpretation of each Operator's flexibility continues 
to include the flexibility to choose curricula, then some consideration should be given to 
identifying measurements that best reflect those curricula for each school. 

Set Realistic Expectations for Indicators of Success in the Early Stages of School 
Conversion 

Individuals at each leadership level learned during the inaugural school year that embedded, 
historical challenges at an ISD school often cannot be overcome in a single year. In addition 
(and as reflected in the Logic Model developed by the evaluation team before the start of the 
initiative; Appendix D), a focus on changes in test scores may lead implementers and observers 
alike to overlook early successes in less easily measurable but equally important formative 
areas (such as community perceptions of changes in school quality). Finally, while most of the 
legislated improvement targets appear to be appropriate (e.g., the expectation that ISD schools 
"exceed the average annual percentage growth of other qualifying schools"), the expectations 
related to the time and resources necessary to reach them (e.g., exceeding the average growth 


15 Some of the provisions in House Bill 798 and Senate Bill 522 may address some (but not all) of the issues 
highlighted in this section. 
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of those schools for at least three consecutive years during the Operator's initial five-year 
contract 16 ) may not be as appropriate. 

Require Operators to Propose and Pursue Bold and Comprehensive Changes 

The changes introduced at Southside Ashpole for Year 1 (extending the school day, introducing 
a new curriculum, requiring uniforms) were notable and noticeable, but it is not yet clear 
whether in the long run they collectively will be sufficient enough to address the school's 
deepest and most persistent challenges. As noted earlier, the process of school reform takes 
time, but whether the pace of change at Southside Ashpole will continue (or increase), or 
whether changes will be limited to those introduced in the first year remains to be seen. As 
demonstrated by several of the other school renewal initiatives across the state in 2018-19 
(such as the Laboratory Schools and the planning work for the School Renewal District), there 
are opportunities for expanding the changes introduced at Southside Ashpole in Year 1 and for 
more comprehensively reimagining how the school could operate. Given the very real 
challenges at Southside Ashpole and the other ISD-eligible schools, significant gains—whether 
academic, social, or otherwise—may depend more on the boldness and depth of the fully- 
realized renewal plans introduced than on the quick resolution of day-to-day challenges that 
sometimes distract schools from focusing on longer-term goals. 

Require Operators to Prepare Annual School-Level Reports 

One of the original goals of the ISD was for its schools to be places for experimentation and 
learning about what works best for each targeted student population. In order to capture these 
unique policies and practices, and for sharing this information more broadly, the evaluation 
team recommended in its original evaluation plan that the ISD require each participating school 
and its Operator to submit an annual report. The first-year operational challenges and initial 
student performance outcomes at Southside Ashpole make the need for such a report even 
greater; without such reports, ISD schools will continue to rely solely on data collected by the 
evaluation team to contextualize their outcomes for each academic year. The evaluation team 
recommends that these reports include information about implementation strategies, 
challenges, and successes, as well as anything else an individual school may opt to include. 
Appendix J outlines specific components to consider for inclusion in the School-level Annual 
Report, aligned to the evaluation questions. 

Increase Pre-Opening Planning Time 

The initiative's annual timeline likely will be an ongoing challenge until it is extended in one or 
more areas. For example. Year 1 implementation at Southside Ashpole would have benefitted 
from a longer planning time (e.g., a full planning year before re-opening), as well as advance 
time to recruit and hire the staff necessary to meet the challenges of a low-performing school. 


16 § 115C-75.12. Term of supervision for an innovative school 

( https://www.ncleg.gov/EnactedLegislation/Statutes/PDF/BySection/Chapter 115C/GS 115C-75.12.pdf ) 
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Similarly, the current school selection timeline (by October 15 each year 17 ) does not allow for 
review of critical current-year data as part of the selection process. 18 

Seek Staff and Partners with Credible Connections to Each ISD School Community 

Some consideration should be given to identifying people and organizations to work in ISD 
settings who bring at least one connecting experience to the work (e.g., previous work in low- 
performing schools or previous work in the impacted community). 19 One administrator posited 
that no one, including the original ISD superintendent, had full knowledge of all of the things 
that needed to be done in order to run the ISD; learning the ins and outs of not only running 
but essentially creating a new district required first experiencing and then finding the time to 
address each challenge as it arose. 


17 § 115C-75.7. Selection of innovative schools 

( https://www.ncleg.gov/EnactedLegislation/Statutes/PDF/BySection/Chapter 115C/GS 115C-75.7.pdf ) 

18 This timeline (along with other operations factors) is addressed in House Bill 798 and Senate Bill 522. 

19 One interviewee noted that, without a local connection, his ISD role sometimes felt like trying to lead in a foreign 
land. 
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Limitations and Next Steps 

Limitations to Qualitative Analyses 

With only one school and one year of implementation data on hand, rigorous quantitative 
analyses of the ISD are not yet possible. Appendix F outlines the quantitative approach that will 
be taken for future reports (should additional schools join the ISD), as well as the limitations 
associated with the procedures described. 

In the meantime, the surveys, interviews, and focus groups conducted during the 2018-19 
school year provided valuable information about Southside-Ashpole that could not be captured 
by quantitative measures. However, three factors limit the conclusions that can be drawn from 
these sources: 

• Potential differences between student and parent survey respondents and non¬ 
respondents; 

• The small number of teachers and administrators; and 

• The lack of multi-year data, including historical data. 

Differences in Student and Parent Survey Respondents and Non-Respondents 

With only a single, small school in the ISD for the 2018-19 school year, there were few students 
and parents available to answer surveys. Only 17 parents, representing 23 of Southside- 
Ashpole's approximately 200 students, chose to take the survey. It is possible that this group of 
parents is not representative of all Southside Ashpole parents (e.g., perhaps these parents are 
more involved in their children's school lives); therefore, their survey responses may not reflect 
the broader opinions of all Southside Ashepole parents. Likewise, only 46 students in 3 rd , 4 th , 
and 5 th grade received the necessary parent permission to take the student survey and were in 
school on the day when the survey was administered. Again, these students may not be 
representative of the entire student body for a number of different reasons. 

Small Size of Teacher, Administrator Samples 

Similar to the first issue, there were only a few teachers and administrators to speak to about 
the ISD. Although almost every teacher and administrator was interviewed and/or surveyed for 
this report, they collectively represent a limited number of voices. 

Lack of Multi-Year Data 

Unlike the quantitative data in the report, which is standard data routinely collected by the 
state each year, most of the qualitative data was collected exclusively for the ISD evaluation. 
With only one year of collection so far, no changes over time are identifiable yet. This limitation 
is compounded by the fact that very few of the teachers and administrators involved with the 
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ISD were sufficiently familiar with Southside Ashpole prior to the ISD to be able to comment 
about changes since the school's conversion. 

Next Steps: School Year 2019-20 Data Collection and Analyses 

The ISD originally was scheduled to incorporate new schools for the 2019-20 school year, but as 
of this writing only the original school continues to be served. Unless changes are proposed by 
the ISD and negotiated as part of a revised Scope of Work, the evaluation team will continue to 
collect qualitative and quantitative data on implementation and outcomes at the current ISD 
school. These data will be used to estimate possible trends as part of the next annual report 
(due Fall 2020). 
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Appendix A: Evaluation Outcomes Required by the Enacting 
Legislation 

The enacting legislation is Session Law 2016-110 (House Bill 1080), 20 which created North 
Carolina General Statute §115C-75 (Article 7A). 21 Outcome measures below were identified in 
§115C-75.11 and §115C-75.12. 

1. School academic growth, performance scores and grades 

a. School overall performance score (target: C or better) 

b. School-level overall growth score (annual) 

c. School-level annual percentage growth (year-to-year) 

d. School-level overall performance scores (ISD vs comparison schools) 

e. School-level overall growth scores (ISD vs comparison schools) 22 

f. School-level mathematics performance and growth by subject area (ISD vs 
comparison schools) 

g. School-level reading performance and growth by subject area (ISD vs comparison 
schools) 

2. Schools establish a safe and positive learning environment 

a. Proportion of students who feel safe at school 

b. Proportion of students with positive perceptions of learning environment 

c. Proportion of staff who feel safe at school 

d. Proportion of staff indicating positive impressions on related Teacher Working 
Conditions Survey items 

e. Proportion of parents who feel school is safe 

f. Proportion of parents who feel that the school is responsive to their concerns 

3. Community engagement grows in ways that are attributable to the ISD 

a. Description of the process of identifying valuable external partners 

b. Annual list of partners and description of services they provide, including level of 
partnerships (time commitment, financial commitment) 

4. Parent engagement grows in ways that are attributable to the ISD 

a. Description of the process of identifying valuable external partners 


20 https://www.ncleg.net/Sessions/2015/Bills/House/PDF/H1080v6.pdf 

21 https://www.ncleg.net/EnactedLegislation/Statutes/HTML/BvArticle/Chapter 115C/Article 7A.html 

22 Note: Legislatively-mandated target: exceeds average growth of all other qualifying schools 
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b. Parent perceptions of the school academic expectations and standards 

c. Parent perceptions of connections between student engagement and academic 
outcomes 

d. Parent perceptions of their own participation in academic life of their students 

5. ISD school staff feelings of empowerment grow in ways that are attributable to the ISD 
school-community engagement efforts 

a. Staff perceptions of empowerment 

b. Parent perceptions of empowerment 

6. ISD school creates an effective structure for holding staff accountable 

a. ISO protocol for teacher evaluation 

b. Staff perceptions of staff accountability process 

7. ISD schools exhibit fiscal efficiency and effectiveness 

a. Annual school operating costs, compared to average operating costs for 
similarly-sized new schools and/or new charter schools 
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Appendix B: Initially Identified Schools, 2017-2019 


Decision Year 


School District School 

2017 2017 Notes 

2018 2018 Notes 

2019 

_ , Hillcrest Elementary 

• 

• Finalist 


Harvey R Newlin Elementary 

• 



Wadesboro Elementary 



• 

Anson County Schools 

Wadesboro Primary 

• 



Buncombe County Schools Johnston Elementary 



• 

Renaissance West STEAM Academy 


• 


Charlotte-Mecklenburg Schools Walter G Byers School 

• 



Bruns Academy 

• Dropped fm consideration; fed aid 



Cumberland County Schools Teresa C Berrien Elementary 



• 

Duplin County Schools Wallace Elementary 

• 



Eno Valley Elementary 

• 

• 


Fayetteville Street Elementary 

• 



Durham Public Schools Eastway Elementary 

• Dropped fm consideration; fed aid 



Glenn Elementary 

• Finalist 



Lakewood Elementary 

• Finalist 



Edgecombe County Public Schools Stocks Elementary 

• 

• 

• 

Ceasar Cone Elementary 

• 



Gillespie Park Elementary 

• 



Guilford County Schools Vandalia Elementary 


• 


Washington Elementary 

• 



Fairview Elementary 


• Finalist 


Harnett County Schools Wayne Avenue Elementary 



• 

..x r. r~ . Ahoskie Elementary 

• 



Riverview Elementary 

• 
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Decision Year 


School District School 

2017 2017 Notes 

2018 2018 Notes 

2019 

Johnston County Schools Selma Middle School 

• 



Lenoir County Schools Northeast Elementary 

• 



East End Elementary 

• 



Martin County Schools Edna Andrews Elementary 

• 



South Creek Elementary 



• 

Nash Rocky Mount Schools Williford Elementary 

• Finalist 

• Finalist 


New Hanover County Schools A H Snipes Academy of Arts & Design 

• 



,, , , , Gaston Middle 

• 

• Finalist 

• 

Willis Hare Elementary 

• Finalist 



„ _ , _ , , Grifton 

• 



South Greenville Elementary 

• 



Orrum Middle 

• 



Rosenwald Elementary 

• 



Robeson County Schools Townsend Middle 

• 



R B Dean Elementary 

• Finalist 



Southside Ashepole Elementary 

• Selected 



Thomasville City Schools Liberty Drive Elementary 

• 



Union County Schools Walter Bickett Elementary 

• 



Brogden Middle 

• Dropped fm consideration; fed aid 

• 

• 

Wnvno Cnimtv «y h nnk Brodgen Primary 

• 



Eastern Wayne Elementary 

• 



Carver Heights Elementary 

• Dropped fm consideration; fed aid 

• Selected; deselected by NCGA 


BO Barnes Elementary 



• 

Wilson County Schools Margaret Hearne Elementary 

• 


• 

Vick Elementary 

• Dropped fm consideration; fed aid 
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Decision Year 


School District School 

2017 2017 Notes 

2018 2018 Notes 

2019 

Ashley Academy 


• 


Diggs-Latham Elementary 

• 



Forest Park Elementary 

• 

• 


Gibson Elementary 

• 



Ibraham Elementary 



• 

Winston-Salem Forsyth County Kimberly Park Elementary 


• 


Sc ^ 00 ' s Middle Fork Elementary 

• 



North Hills Elementary 

• 



Old Town Elementary 

• 


• 

Ashley Academy 

• Droppedfm consideration; fed aid 



Flail-Woodward Elementary 


• Finalist 


Kimberly Park Elementary 

• Dropped fm consideration; fed aid 
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Appendix C: Summary of Relevant Pending Legislation 

Proposed Changes to ISD Legislation in H798 v2 23 

House Bill 798 (2019-2020 Session) proposes several changes to the existing ISD legislation. 

Summary of Major Changes 

• The criteria for inclusion in the ISD are revised: The requirement that ISD schools be 
elementary schools is eliminated (i.e., middle and high schools also would be eligible). 

• The process for selecting an innovative school is updated: Schools would go through a 
three-year process from the time of qualification to selection for the ISD, with multiple 
mandated public hearings during that period. The ISD Superintendent would be 
required to help qualifying schools make improvements (and thus no longer be eligible 
for consideration) during that time. 

• The management options for innovative schools are expanded: Selected schools would 
now get either an IS operator or an IS consultant (determined by the State Board of 
Education [SBE]); if they receive the latter, the school would continue to be operated by 
the LEA and it must meet requirements established by the ISD and the consultant. 

• The responsibility for certain decisions is shifted: IS operators would be required to work 
in collaboration with (rather than independently or in consultation with) the ISD 
Superintendent on several major decisions, including the hiring and firing of principals. 

Additional Details 

The legislation amends the criteria for qualification so that schools meeting at least one of the 

following will begin the evaluation process: 

• Title I school in the lowest 5% of all Title I schools; 

• School serving students in grades 9-12 that failed to graduate a third or more of 
students; or 

• School identified by the SBE as "being in need of comprehensive support and 
improvement" due to at least one underperforming subgroup. 

The evaluation process (currently called the selection process) becomes a three-year period 

during which schools are evaluated and the public is updated annually: 


23 https://www.ncleg.gov/BNILookUp/2019/H798 
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• In the first year after a school has been identified as qualifying, the school will be placed 
on the ISD qualifying list; the ISD Superintendent will hold a public hearing for parents, 
school employees, and community members. 

• If a school on the qualifying list in the prior year remains a qualifying school in the next 
year, the school will be placed on the ISD watch list; the local Board of Education will 
hold a public hearing. 

• If a school on the watch list in the prior year remains a qualifying school in the next year, 
the school will be placed on the ISD warning list. The local Board of Education will hold a 
public hearing and present at a local public county commissioners meeting. 

• If a school on the warning list remains a qualifying school in the next year, and is one of 
the lowest five qualifying schools on the warning list, the school will be selected by the 
SBE as an innovative school, beginning the next year. 

The proposed legislation removes the requirements that the ISD Superintendent is responsible 
for recommending schools for inclusion in the ISD, that the selected schools each year 
represent geographic diversity, and that there is no more than one school per LEA selected 
each year. 

The SBE will ensure that schools on any list are "engaged in strategies ... for comprehensive 
support and improvement." The ISD Superintendent will assist local Boards of Education in 
"identifying funding, strategies, and partners for the comprehensive support and improvement 
efforts." 

The SBE can select either an IS operator or an IS consultant to manage the innovative school. 
The Department of Public Instruction can be selected as an operator, but the ISD itself cannot. 

If the school is assigned an IS consultant, the school will continue to be operated by the LEA, 
but it will be obligated to meet requirements set by the ISD Superintendent and the consultant 
for a minimum of five years; if it fails to meet the requirements, it will be assigned an IS 
operator by the SBE. 

IS operators must work in collaboration with the ISD Superintendent—not independently or in 
consultation with the ISD Superintendent—to make decisions about: a) hiring and removing the 
school's principal; and b) entering into MOUs with the local Board of Education to address 
facility and capital expenditures, transportation, and services for children with disabilities. 

Proposed Changes to ISD Legislation in S522 \ /6 U 

Significant Differences between H798 and S522 


24 https://www.ncleg.gov/BNILookUp/2019/S522 
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House Bill 798 and Senate Bill 522 are similar in many ways; however, there are a few notable 
differences: 

Text 

Text 

Text 
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Appendix D: Original Logic Model 


Inputs 

(resources, activities) 


Operator with history 
of success 


State-vetted plan for 
school turnaround 


State support (via 
ISD superintendent) 


Equitable funding 
(100% of LEA state 
and local PPE) 


Charter-like flexibility 


Professional learning 
opportunities provided 
by turnaround 
operator 


Identification of 
academic and 
non-academic 
barriers 



Identification and 
dissemination of 
effective 
leadership 
practices 


Identification 

and 

dissemination of 
best-practices 
strategies 
through partner 
organizations 


Identification of 
potential 
community 
partners & 
community 
leaders 


All teachers 
participate in 
ISD-related 
professional 
learning 
opportunities 


Changes in school-community 
engagement 


Immediate changes in 
approaches to leadership 



Outcomes 

Shorter-Term Outcomes 


Longer-Term Outcomes 

Changes in learning 
conditions (including changes 
in student behavior) 

Changes in participating 
LEA’s & state s culture of 
schooling 



Immediate changes in 
sustainable, high-quality 
staffing 

Changes in student and 
school academic growth and 
achievement 


Longer-term changes in 
sustainable, high-quality 
staffing 


Longer-term changes in 
approaches to leadership 
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Appendix E: Crosswalk: Evaluation Questions, Measurable Outcomes, Indicators, and Data 
Sources 


Evaluation Question 

Measurable Outcome 

Indicator(s) 

Data Source(s) 

Ql: Does the ISD 

ISD schools achieve and 

School overall performance score (target: C or better) 

North Carolina 

improve student- and 
school-level academic 

maintain performance scores at 
or above a C 

(Note: Overall combined measure of achievement and growth) 

administrative data 

growth and 

ISD students demonstrate 

Student cohort proficiency, by grade, subject area, and sub-group 

North Carolina 

achievement ? 

academic proficiency 


administrative data 


ISD schools demonstrate 

School-level overall growth score (annual) 

North Carolina 


academic growth 

School-level annual percentage growth (year-to-year) 

administrative data 


ISD schools' academic 

School-level overall performance scores (ISD vs comparison 

North Carolina Report 


outcomes compare favorably to 
eligible but non-identified 
schools 

schools) 

School-level overall growth scores (ISD vs comparison schools) 

(Note: Legislatively-mandated target: exceeds average growth 
of all other gualifying schools) 

School-level mathematics performance and growth by grade and 
sub-group (ISD vs comparison schools) 

School-level reading performance and growth by grade and sub¬ 
group (ISD vs comparison schools) 

Card data 


ISD students exhibit more 
growth than students at 
matched a) local schools 
(possibly including 1-Zone 
schools), b) transformational 
schools (e.g., Restart schools), 
and c) considered but non- 
selected schools; also vs 
statewide growth averages 

Student cohort growth score, by grade, subject area, and sub-group 

North Carolina 

administrative data 


Key: Red text= measurable outcome for evaluation elements highlighted in enacting legislation; S7=Short term outcome (1-3 years); iT=Long term outcome (4+ 
years) 
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Evaluation Question 

Measurable Outcome 

Indicator(s) 

Data Source(s) 

Q2. Does the ISD 
improve learning 
conditions, including 
changes in student 
behavior? 

ISD student support services 
(Exceptional Children services, 
etc.) meet statutory 
requirements 

Description of EC plan (outlining the full continuum of services to 
meet the special education and related services needs of students 
with disabilities) 

Proof of a highly-qualified and licensed special education teacher(s) 
and/or director 

Identification of sources for external EC service providers (e.g., 
school psychologist, occupational therapist, physical therapist, 
speech language pathologist, audiologist, etc.) 

Evidence of on-site/accessible, appropriate instructional supports 
for students with disabilities (as indicated in lEPs) 

Statement of school policies for EC, incorporating all required 
elements to meet Federal and State regulations (e.g., discipline, 
confidentiality, accountability, maintenance of effort, security and 
confidentiality of EC Student Special Education files, etc.) 

Address transportation (as a related service) for EC students who 
may have this as a component of their IEP 

Annual school report 

ISD schools establish a safe and 
positive learning environment 

Safe 

Proportion of students who feel safe at school 

Positive 

Proportion of students with positive perceptions of learning 
environment 

Student survey 

Safe 

Proportion of staff who feel safe at school 

Positive 

Proportion of staff indicating positive impressions on related 
Teacher Working Conditions Survey items 

NC Teacher Working 
Conditions Survey 

Safe 

Proportion of parents who perceive that school is safe 

Positive 

Proportion of parents who feel that the school is responsive to 
their concerns 

Parent survey 


Key: Red text= measurable outcome for evaluation elements highlighted in enacting legislation; ST=Short term outcome (1-3 years); iT=Long term outcome (4+ 
years) 
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Evaluation Question 

Measurable Outcome 

Indicator(s) 

Data Source(s) 

Q2 (cont.). Does the 

ISD improve learning 
conditions, including 
changes in student 
behavior? 

ISD schools experience 
reduction of behavioral referrals 

Description of school's approach to discipline and changes over 
time 

School-level annual 

report 

Counts of disciplinary infractions and suspensions, by demographic 
group, including changes over time 

Counts of disciplinary infractions and suspensions, relative to 
previous/feeder school 

North Carolina 

administrative data 

ISD schools experience 
reduction of student 
absenteeism and/or increased 
attendance 

Description of school's responses to attendance problems/truancy 

School-level annual 
report 

Daily attendance rate and count of chronically absent students, 
including changes over time 

Daily attendance rate and count of chronically absent students, 
relative to previous/feeder school 

North Carolina 

administrative data 

Q3. Does the ISD 
contribute to changes 
in school-community 
engagement ? 

Community engagement grows 
in ways that are attributable to 
the ISD 

• Formal relationships are 
established between ISD 
schools and community 
representatives 

• In-kind supports and services 
provided to ISD schools 
change in positive ways 

Description of the process of identifying valuable external partners 

Annual list of partners and description of services they provide, 
including level of partnerships (time commitment, financial 
commitment) 

If evaluation budget supports: Descriptive report of offsetting 
services or additional value in services provided; where available, 
budget data on any financial offsets arising from partnerships 

School-level annual 

report 

Parent engagement grows in 
ways that are attributable to 
the ISD 

Annual list of parent/guardian involvement in school events 

School-level annual 

report 


Key: Red text= measurable outcome for evaluation elements highlighted in enacting legislation; S7=Short term outcome (1-3 years); iT=Long term outcome (4+ 
years) 

The William and Ida Friday Institute for Educational Innovation 48 
















Innovative School District: Year 1 Evaluation 


DRAFT - September 2019 


Evaluation Question 

Measurable Outcome 

Indicator(s) 

Data Source(s) 

Q3 (cont.). Does the 

ISD contribute to 
changes in school- 
community 
engagement ? 

• ISD parents/guardians 
understand academic 
expectations and standards 

• ISD parents/guardians 
understand connections 

between student 
engagement and academic 

outcomes 

• ISD parents/guardians 
participate more frequently 
in the life of the school 

• ISD parents/guardians 
participate in the academic 
life of their own students 

Parent perceptions of the school academic expectations and 
standards 

Parent perceptions of connections between student engagement 
and academic outcomes 

Parent perceptions of their own participation in academic life of 
their students 

Parent survey 

ISD school staff and community 
member feelings of 
empowerment grow in ways 
that are attributable to the ISD 
school-community engagement 
efforts 

Staff perceptions of empowerment 

Practitioner survey 
Practitioner focus group 

Parent perceptions of empowerment 

Parent survey 


Q4. Does the ISD contribute to changes in the culture of schooling both in and outside of the ISD? 


Q4a. Does the ISD 
contribute to an 

ISOs lead diffusion of best 
practices across ISD schools 

Plan for diffusion of best practices 

School-level annual 

report 

overall change in 

the culture of 
schooling in ISD 
schools? 


Staff perceptions of ISO-provided guidance for implementation of 
best practices 

Practitioner survey 

Q4b. Does the ISD 
change approach¬ 
es to ensuring 

The ISD implements process to 
build a local, sustainable 
teacher workforce pipeline (LT) 

Description of teacher recruitment and retention processes 

School-level annual 

report 

sustainable, high 

ISD schools recruit effective 

State teacher evaluation data 

North Carolina 

quality staffing in 
ISD schools ? 

staff (LT) 

Teacher absentee rates, including chronic absentee episodes and 
changes over time 

administrative data 
(NCEES) 


Key: Red text= measurable outcome for evaluation elements highlighted in enacting legislation; S7=Short term outcome (1-3 years); iT=Long term outcome (4+ 
years) 
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Evaluation Question 

Measurable Outcome 

Indicator(s) 

Data Source(s) 

Q4b (cont.). Does 
the ISD change 
approaches to 
ensuring 
sustainable, high 
quality staffing in 
ISD schools ? 

ISD schools retain effective staff 
(LT) 

Staff turnover rate, including changes over time 

Staff turnover rates (ISD vs comparison schools) 

North Carolina 

administrative data 

ISD school creates an effective 
structure for holding staff 
accountable (ST) 

ISO protocol for teacher evaluation 

School-level annual 

report 

Staff perceptions of staff accountability process 

[Note: Accountability default: state evaluation process) 

Practitioner survey 
Practitioner focus group 

The ISD promotes a culture of 
professional learning among 
teachers (ST) 

Annual list of professional development opportunities (plus 
attendance figures) 

School-level annual 

reports 

Staff perceptions of school-level culture of professional learning 

Practitioner surveys 
Practitioner focus group 

Q4c. Does the ISD 
change 

approaches to 
leadership in ISD 
schools ? 

ISD schools are led by effective 
school administrators (ST) 

(ISD to define in partnership with the evaluator effective 
leadership and related measurement; default: North Carolina 
administrator evaluation tool) 

North Carolina 
administrative data (NC 
administrator evaluation 
data) 

Parent perceptions of school administration leadership 

Parent Survey 

Staff perceptions of school administration leadership 

Practitioner survey 
Practitioner focus group 

ISD schools adopt effective 
leadership strategies (ST) 

(ISD to define in partnership with the evaluator; see above) 

School-level annual 

report 

(ISD to define in partnership with the evaluator; see above) 

Practitioner survey 
Practitioner focus group 

(ISD to define in partnership with the evaluator; see above) 

Parent Survey 

ISD schools exhibit fiscal 
efficiency and effectiveness (ST) 

Annual school operating costs, compared to average operating 
costs for similarly-sized new schools and/or new charter schools 

North Carolina 

administrative data 

School-level annual 
report (annual budget 
report) 

ISD schools establish a diffused 
leadership model that includes 
teacher leadership (LT) 

Description of teacher leadership model 

Evidence of teacher leadership model implementation 

School-level annual 

report 

Staff perception of the teacher leadership model 

Practitioner survey 
Practitioner focus group 


Key: Red text= measurable outcome for evaluation elements highlighted in enacting legislation; ST=Short term outcome (1-3 years); LT=Long term outcome (4+ 
years) 

The William and Ida Friday Institute for Educational Innovation 50 































Innovative School District: Year 1 Evaluation 


DRAFT - September 2019 


Evaluation Question 

Measurable Outcome 

Indicator(s) 

Data Source(s) 

Q4d. Does the ISD 
contribute to 
changes in the 

culture of 
schooling in 
partnering LEAs, 
non-participating 
LEAs, and/or 
across the State 

overall ? 

ISD leadership encourage 
diffusion and replication of 
strategies identified as effective 
practices to partnering LEAs 

Plan for diffusion of best practices to partner LEAs 

School-level annual 

report 

Actions taken by partner LEAs that are attributable to ISD efforts 

Interviews with 
neighboring LEA 
staff/superintendents 

Non-participating LEAs 
demonstrate leadership-level 
changes in approaches to 
school leadership 

Actions taken by non-partnering but originally eligible schools 
(principals) reflect best practices identified by the ISD 

Actions taken by non-partnering but originally eligible LEAs 
(superintendents and local school boards) reflect best practices 
identified by the ISD 

Document/policy scan of 
the eligible but non¬ 
participating 
schools/LEAs 


Note on pre-intervention data: When possible, use 2016-17 data for pre-intervention data baseline, since these are the data upon 
which Ashpole was selected; similar logic for data upon which next cohort of schools were selected. 


Key: Red text= measurable outcome for evaluation elements highlighted in enacting legislation; S7=Short term outcome (1-3 years); iT=Long term outcome (4+ 
years) 
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Appendix F: Comparison School Analysis Procedure 


Given the quantitative analysis limitations described in the main body of this report and limits 
imposed by the evaluation budget, the evaluation development team hosted a quantitative 
analysis summit in April 2018 with independent experts to discuss reasonable quantitative 
options for this evaluation. Based on that consultation, the evaluation development team 
proposed the following recommendations for the quantitative analysis component of the 
evaluation; the evaluation team elected to follow these recommendations. 

Comparative Interrupted Time Series 

Because of such limiting factors as the small size of the ISD initiative, summit participants 
recommended an interrupted time series design for the final quantitative analyses. To prepare 
for this analysis, the evaluator collected multiple quantitative indicators for every school in the 
state for the four years before the first school was transferred into the ISD. In addition, the 
evaluator collected the same indicators for the first year of operation of the first school and will 
continue to collect these same indicators through the end of the evaluation. Eventually (once a 
school has been in ISD for at least two but preferably three years), the main analyses will 
examine whether there were any changes in trend lines for the key student and teacher 
outcomes after ISD inclusion. To strengthen this design, the evaluator will include comparison 
groups, making it a comparative interrupted time series. 

Because the interrupted time series analysis relies on analyzing changes over time, the 
evaluation team initially will conduct difference-in-differences analyses (which, though not as 
compelling in terms of their ability to model changes over time, do allow for single-year 
analyses) for the first two outcome years (the Baseline and Growth years). 

Comparison Schools 

The summit participants identified four potential groups from which comparison schools could 
be selected. As noted in the North Carolina Administrative Data section, above, the first 
includes the next-in-line schools that were announced publicly as being eligible for ISD 
inclusion, but ultimately were not selected (Appendix B). The second group includes schools 
that were not eligible for ISD inclusion but are similar to the schools selected for the ISD on 
multiple relevant measures (Appendix G). The third group includes other schools from the ISD 
school's original LEA, when such comparisons are relevant. The final potential comparison 
group includes any schools selected for an 1-Zone within the participating LEA. The analyses 
included in the first reports for this evaluation use the first two groups only (the original school 
district of the first participating school opted not to petition to run other schools in an 1-Zone). 
The full list of comparison schools is included in Table FI (following page), but the identity of 
the two statistically-masked comparison schools has been masked. 
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Table FI. Historical Academic Outcomes Data for Inaugural ISD School and Comparison Schools 


ID 

Name 

LEA 

2014-15 

2015-16 

2016-17 

2017-18 

2018-19 

2014-15 

2015-16 

2016-17 

2017-18 

2018-19 

2014-15 

2015-16 

2016-17 

2017-18 

2018-19 

ISD Schools 




780408 

Southside Ashpole Elementary 

Robeson 

F 

36 

F 

35 

F 

27 

F 

29 

F 

30 

29.4 

23.6 

18.4 

20 

20.5 

NotMet 

Met 

NotMet 

NotMet 

NotMet 

Statistically-Matched Elementary Schools 




340368 

Matched School A 

LEA A 

F 

29 

F 

33 

F 

33 

F 

39 

F 

36 

17.5 

20.7 

22.1 

28.8 

26.6 

Met 

Met 

Met 

Met 

Met 

800346 

Matched School B 

LEA B 

F 

36 

D 

40 

F 

29 

F 

25 

F 

26 

31 

34.5 

22.2 

18.1 

17.1 

NotMet 

NotMet 

NotMet 

NotMet 

NotMet 

ISD Finalist Schools 




320320 

Glenn Elementary 

Durham 

F 

39 

F 

36 

F 

37 

D 

50 

D 

42 

29.4 

30 

32.3 

41 

35.4 

Met 

NotMet 

NotMet 

Met 

NotMet 

320339 

Lakewood Elementary 

Durham 

F 

38 

F 

35 

F 

35 

F 

37 

C 

55 

29.5 

26.9 

28.2 

28.8 

46.1 

Met 

NotMet 

NotMet 

Met 

Exceeded 

640396 

Williford Elementary 25 

Nash Rocky Mount 

D 

41 

F 

37 

F 

26 

F 

30 

F 

34 

31 

27.4 

19 

19.6 

* 

Met 

Met 

NotMet 

NotMet 

* 

660360 

Willis Hare Elementary 

Northampton 

D 

49 

D 

48 

F 

36 

D 

51 

D 

49 

44.5 

43.1 

31.1 

43.8 

41.6 

NotMet 

NotMet 

NotMet 

Met 

Met 

780390 

R B Dean Elementary 26 

Robeson 

F 

34 

F 

36 

F 

36 

D 

40 

D 

43 

23.1 

29.3 

29.5 

29.4 

33.9 

Met 

NotMet 

NotMet 

Met 

Met 

Other Elementary Schools on ISD Selection List 




10354 

Harvey R Newlin Elementary 

Alamance-Burlington 

D 

40 

D 

42 

F 

37 

D 

42 

D 

43 

28.8 

32 

30.7 

34.8 

34.5 

Met 

Met 

NotMet 

Met 

Met 

10358 

Hillcrest Elementary 

Alamance-Burlington 

D 

50 

D 

47 

F 

34 

F 

34 

D 

42 

41.7 

40.6 

29.8 

28 

34.9 

Met 

Met 

NotMet 

NotMet 

Met 

40330 

Wadesboro Primary 

Anson 

F 

36 

F 

39 

D 

43 

D 

40 

F 

37 

26.3 

28 

33.2 

29.3 

29.1 

Met 

Met 

Met 

Met 

NotMet 

292316 

Liberty Drive Elementary 

Thomasville City 

D 

52 

D 

53 

D 

42 

D 

47 

D 

41 

47.3 

48.5 

38.6 

38.7 

36.7 

Met 

Met 

NotMet 

Met 

NotMet 

310388 

Wallace Elementary 

Duplin 

D 

49 

D 

47 

D 

41 

D 

52 

D 

47 

44.1 

43.6 

35.8 

42.3 

42.3 

NotMet 

NotMet 

NotMet 

Exceeded 

NotMet 

320315 

Eno Valley Elementary 

Durham 

F 

37 

F 

37 

F 

36 

F 

38 

D 

50 

27.9 

29.8 

25.5 

29.4 

40.7 

Met 

NotMet 

Met 

Met 

Exceeded 

320344 

Fayetteville Street Elementary 

Durham 

D 

43 

D 

44 

F 

38 

D 

46 

D 

40 

33.1 

36.6 

32.4 

37.6 

30.1 

Met 

Met 

NotMet 

Met 

Met 

330354 

Stocks Elementary 

Edgecombe 

F 

30 

F 

33 

F 

32 

F 

34 

F 

37 

21.8 

23.8 

20.5 

23 

28.4 

NotMet 

NotMet 

Met 

Met 

NotMet 

340376 

Forest Park Elementary 

Winston-Salem Forsyth 

D 

40 

F 

32 

F 

29 

F 

33 

D 

41 

34 

26.6 

17.2 

23.5 

29.7 

NotMet 

NotMet 

Met 

Met 

Met 

340380 

Gibson Elementary 

Winston-Salem Forsyth 

F 

37 

F 

30 

F 

35 

D 

45 

D 

54 

28.4 

24.7 

24.8 

34.6 

44.8 

Met 

NotMet 

Met 

Met 

Exceeded 

340430 

Diggs-Latham Elementary 

Winston-Salem Forsyth 

F 

36 

D 

44 

F 

39 

D 

41 

D 

43 

29.9 

38.2 

33.6 

36.9 

36.9 

NotMet 

NotMet 

NotMet 

NotMet 

NotMet 

340447 

Middle Fork Elementary 27 

Winston-Salem Forsyth 

F 

33 

F 

31 

F 

34 

F 

30 

D 

40 

27.1 

23.5 

22.9 

22.1 

28.9 

NotMet 

NotMet 

Met 

NotMet 

Met 

340462 

North Hills Elementary 

Winston-Salem Forsyth 

D 

42 

D 

49 

F 

37 

F 

39 

D 

48 

33.2 

41.8 

31 

33.8 

37.7 

Met 

Met 

NotMet 

NotMet 

Exceeded 

340476 

Old Town Elementary 

Winston-Salem Forsyth 

D 

43 

D 

49 

D 

42 

D 

43 

F 

39 

39 

42 

37.5 

37.8 

34.7 

NotMet 

Met 

NotMet 

NotMet 

NotMet 

410349 

Ceasar Cone Elementary 

Guilford 

F 

33 

F 

36 

F 

34 

F 

39 

F 

36 

21.8 

27.2 

25.3 

29.4 

29.1 

Met 

NotMet 

NotMet 

Met 

NotMet 

410385 

Gillespie Park Elementary 

Guilford 

F 

29 

F 

37 

F 

33 

D 

43 

D 

40 

21.8 

26.5 

24.6 

31.3 

32.1 

NotMet 

Met 

NotMet 

Exceeded 

Met 

410586 

Washington Elementary 

Guilford 

F 

38 

D 

41 

D 

41 

F 

39 

D 

44 

30.8 

31.8 

32.1 

28.5 

32.7 

NotMet 

Met 

Met 

Met 

Exceeded 

460308 

Ahoskie Elementary 

Hertford 

D 

42 

D 

47 

D 

41 

D 

40 

F 

39 

35.3 

40.1 

36.3 

37.5 

36.4 

NotMet 

Met 

NotMet 

NotMet 

NotMet 

460332 

Riverview Elementary 

Hertford 

D 

40 

D 

46 

D 

42 

D 

49 

D 

51 

30.2 

40.1 

37 

46.2 

49.2 

Met 

Met 

NotMet 

NotMet 

NotMet 

540325 

Northeast Elementary 

Lenoir 

F 

34 

D 

40 

D 

42 

D 

47 

D 

42 

25.7 

32.9 

31.8 

39.1 

33.8 

NotMet 

NotMet 

Met 

Met 

Met 

580316 

East End Elementary 28 

Martin 

F 

33 

F 

37 

D 

40 

D 

48 

F 

39 

25 

26 

30.5 

40.2 

33.1 

NotMet 

Met 

Met 

Met 

NotMet 

580320 

Edna Andrews Elementary 29 

Martin 

D 

41 

D 

44 

F 

35 

F 

35 



33.5 

35.7 

27.6 

24.2 








25 Reduced grade span from PK-5 to PK-2 (2018-19) 

26 Merged with Townsend Middle School; grade span extended to PK-8 (2019-20) 

27 Converted to Lab School (ID=34Z000; 2018-19) 

28 Merged with Edna Andrews and changed name to South Creek (2018-19) 

29 Merged with East End to become South Creek (2018-19) 
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ID 

Name 

LEA 

2014-15 

2015-16 

2016-17 

2017-18 

2018-19 

2014-15 

2015-16 

2016-17 

2017-18 

2018-19 

2014-15 

2015-16 

2016-17 

2017-18 

2018-19 

600574 

Walter G Byers School 

Charlotte-Mecklenburg 

F 

36 

F 

39 

F 

38 

D 

45 

D 

52 

27.8 

29.5 

28.1 

34.2 

42.9 

NotMet 

Met 

Met 

Exceeded 

Exceeded 

650384 

A H Snipes Acad of Arts/Design 

New Hanover 

F 

39 

D 

41 

F 

35 

D 

41 

D 

43 

29.6 

32.2 

27.5 

32.8 

34.8 

Met 

Met 

NotMet 

Met 

Met 

740352 

Grifton 

Pitt 

D 

42 

D 

46 

F 

34 

D 

46 

D 

54 

31.7 

37.1 

28.5 

34.8 

46.8 

Met 

Met 

NotMet 

Exceeded 

Met 

740390 

South Greenville Elementary 

Pitt 

F 

35 

D 

41 

D 

41 

F 

38 

F 

34 

26.8 

31.3 

30.6 

26.8 

25 

NotMet 

Met 

Met 

Met 

Met 

780394 

Rosenwald Elementary 

Robeson 

F 

33 

F 

28 

F 

35 

D 

45 

D 

43 

23.8 

19.3 

25.3 

34.8 

32.7 

Met 

NotMet 

Met 

Exceeded 

Met 

900370 

Walter Bickett Elementary 

Union 

F 

39 

D 

46 

F 

38 

D 

50 

D 

54 

33.7 

36.6 

32.6 

39.7 

45.8 

NotMet 

Met 

NotMet 

Exceeded 

Exceeded 

960314 

Brodgen Primary 

Wayne 

F 

38 

D 

48 

D 

42 

D 

42 

D 

48 

29.3 

38.9 

35.6 

34 

37.4 

Met 

Met 

NotMet 

Met 

Exceeded 

960329 

Eastern Wayne Elementary 

Wayne 

D 

46 

D 

48 

D 

41 

D 

48 

D 

49 

36.8 

39.7 

36.3 

38.9 

42.4 

Met 

Met 

NotMet 

Met 

Met 

980356 

Margaret Hearne Elementary 

Wilson 

D 

45 

D 

45 

F 

37 

D 

54 

F 

35 

35.5 

38.4 

30 

47 

30.4 

Met 

Met 

NotMet 

Met 

NotMet 

Other Robeson County Elementary Schools 




780320 

Deep Branch Elementary 

Robeson 

F 

34 

D 

47 

D 

51 

D 

53 

D 

53 

27.2 

37.6 

42.8 

45.9 

45.1 

NotMet 

Met 

Met 

Met 

Met 

780322 

East Robeson Elementary 

Robeson 

B 

71 

B 

74 

B 

77 

B 

78 

A 

85 

65.9 

68.3 

73.4 

73.7 

82.7 

Exceeded 

Exceeded 

Exceeded 

Exceeded 

Exceeded 

780328 

Green Grove Elementary 

Robeson 

C 

58 

C 

61 

C 

55 

C 

64 

C 

66 

52 

57.1 

49.1 

59.8 

62.9 

Met 

Met 

Met 

Met 

Met 

780329 

J C Hargrove Elementary 30 

Robeson 

F 

36 

D 

44 

D 

44 

D 

51 

D 

45 

25.3 

34.2 

37.2 

42.9 

35.3 






780340 

Long Branch Elementary 

Robeson 

D 

46 

D 

46 

D 

49 

C 

57 

C 

59 

36.6 

37.1 

41.3 

49.7 

53.3 

Met 

Met 

Met 

Exceeded 

Met 

780344 

Magnolia Elementary 

Robeson 

F 

35 

F 

37 

D 

43 

D 

44 

C 

59 

26.9 

29.1 

35.3 

36 

49.3 

NotMet 

NotMet 

Met 

Met 

Exceeded 

780356 

Oxendine Elementary 

Robeson 

D 

42 

D 

44 

D 

42 

D 

53 

D 

53 

34 

33.2 

35 

42.8 

44.7 

Met 

Exceeded 

Met 

Exceeded 

Exceeded 

780360 

Parkton Elementary 

Robeson 

D 

49 

D 

53 

D 

52 

D 

53 

D 

41 

41 

44.2 

46.3 

46 

36.8 

Met 

Exceeded 

Met 

Met 

NotMet 

780364 

Pembroke Elementary 

Robeson 

D 

42 

D 

52 

D 

45 

D 

51 

D 

53 

35.5 

41.8 

38.6 

42.9 

47.3 

NotMet 

Exceeded 

NotMet 

Exceeded 

Met 

780374 

Peterson Elementary 

Robeson 

F 

38 

D 

43 

F 

34 

D 

43 

D 

44 

27.8 

32.4 

28.3 

36.6 

39 

Met 

Exceeded 

NotMet 

NotMet 

NotMet 

780376 

Piney Grove Elementary 

Robeson 

D 

46 

C 

55 

D 

51 

D 

48 

D 

41 

40.7 

49.6 

49 

45.4 

38.2 

NotMet 

Met 

NotMet 

NotMet 

NotMet 

780384 

Prospect Elementary 

Robeson 

D 

49 

D 

48 

D 

48 

D 

53 

C 

59 

42.4 

44.2 

43.8 

45.6 

51.4 

Met 

NotMet 

NotMet 

Met 

Exceeded 

780392 

Rex-Rennert Elementary 

Robeson 

D 

43 

F 

34 

F 

35 

D 

40 

D 

40 

30.7 

24.6 

25.6 

27.5 

28.4 

Exceeded 

Met 

Met 

Exceeded 

Met 

780398 

Rowland Norment Elementary 

Robeson 

D 

51 

C 

64 

C 

58 

C 

60 

C 

61 

43.5 

58.3 

50.8 

54.4 

56.1 

Met 

Exceeded 

Exceeded 

Met 

Met 

780400 

St Pauls Elementary 

Robeson 

D 

45 

D 

46 

D 

47 

D 

49 

D 

41 

32.5 

35.2 

37.6 

40.2 

35.9 

Exceeded 

Exceeded 

Met 

Met 

NotMet 

780407 

Tanglewood Elementary 

Robeson 

C 

65 

B 

76 

B 

73 

B 

78 

B 

73 

61.5 

72.8 

70.6 

78 

70.1 

Met 

Exceeded 

Met 

Met 

Met 

780412 

Union Chapel Elementary 

Robeson 

D 

50 

D 

49 

D 

47 

D 

54 

D 

45 

41.7 

40.4 

41.6 

44.3 

39.4 

Exceeded 

Met 

NotMet 

Exceeded 

NotMet 

780416 

Union Elementary 

Robeson 

C 

57 

D 

52 

D 

54 

C 

59 

C 

59 

48.2 

47.7 

46.3 

50 

51.8 

Exceeded 

Met 

Met 

Exceeded 

Exceeded 

780417 

W H Knuckles Elementary 31 

Robeson 

F 

36 

F 

37 

F 

38 

D 

44 

D 

45 

24.7 

27 

27.9 

34 

36.1 

Met 

Met 

Met 

Exceeded 

Met 


30 Merged with W H Knuckles; reduced grade span from PK-4 to PK-3 (2019-20) 

31 Merged with J C Hargrave; reduced grade span from PK-4 to PK-3 (2019-20) 
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Limitations to Quantitative Analyses 

Per the enacting legislation, there is high demand for a defensible, quantitative assessment of 
the direct impact of the initiative on student and teacher outcomes. There are, however, at 
least four factors that significantly limit the strength of the quantitative analyses being 
conducted for this evaluation: 

• The small number of schools included in the initiative; 

• Potential differences across school governance in the ISD; 

• The initiative timeline; and 

• The likely lack of randomization of impacted students and educators. 

Number of Schools 

The ISD selected only one school for the 2018-19 school year, and no schools will be brought 
into the ISD for the 2019-20 school year. Measures of changes in teacher behavior, teacher 
quality, and student outcomes are reportable (as we have begun to do in this report), but with 
teachers and students distributed across only a handful of schools by the third year of the 
program—and most in schools that will have been involved with ISD for only one year—results 
of the statistical analyses of these measures must be reviewed with extreme caution. The 
challenge is compounded by the second and third issues below. 

Differences across ISD School Governance 

By statute, schools in the ISD can be governed by different ISOs, meaning that there are likely to 
be substantial differences in school-level goals and strategies, once more schools are brought 
into the ISD. If schools are operated in vastly different ways, combining data from multiple ISD 
schools in an attempt to conduct stronger analyses of impacts on larger groups of teachers and 
students may be advisable only for estimating the overall impact of the presence of the policy— 
not for estimating the impact of different implementations of the policy across sites. 

Initiative Timeline 

At its heart, the ISD is about creating new school cultures. We know from studies of changes in 
even single school culture variables (for example, changes in principal leadership) that schools 
often experience a regression in outcomes for at least a year before even a highly successful 
program begins to show positive results. 

Randomization 

The characteristics of students who attend ISD schools will differ across schools. Furthermore, 
the ISD does not determine student admission by some form of randomization (e.g., via 
lottery). Since randomization of admission is unlikely, direct comparisons of student and 
teacher outcomes are likely to be misleading as a result of the different motivations of students 
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and teachers who choose to learn or teach in ISD schools, both relative to one another and 
relative to traditional public schools. None of these factors prevents evaluators from 
determining meaningful and/or statistically significant correlations between program initiative 
factors and outcomes of interest, but it does prevent evaluators from identifying any causal 
links between program characteristics and student and educator outcomes. 
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Appendix G: Comparison Matching Procedures 


School Matching and ITS 

An important key to a strong non-experimental analysis design is identification of a comparison 
group of non-impacted entities (in educational research, usually schools or individuals) that 
most closely resembles the group of impacted entities, to reduce what is known as selection 
bias. Selection bias occurs when the impacted entities take part in the intervention for one or 
more (often unseen) shared reasons that may themselves be the cause of differences between 
outcomes for that group and outcomes for the comparison group—not the cause of 
participation in the initiative being studied. In other words, "[differences in outcomes between 
the treatment and comparison group may be due to pre-existing or unobserved differences 
between the two groups, rather than to the effect of the program being evaluated" (Somers et 
al. 2013, p. I). 32 

With only about 2,600 schools in North Carolina, and with the constant background noise of 
multiple, overlapping, and sometimes conflicting initiatives in operation in any of them at any 
given time, it can be challenging to identify a reasonable comparison group of schools to help 
strengthen the analyses of outcomes for the subset of schools impacted by a given policy—in 
this case, impacted by the introduction of the ISD. In addition, in North Carolina there is the 
added challenge of identifying whether a given school—whether an ISD school or a potential 
comparison school—and its staff have been exposed to similar programs in the recent past. For 
example, in recent years, many of the schools on the ISD consideration list (Appendix B) have 
been impacted directly or indirectly by the work of the state District and School Transformation 
team, which was tasked with identifying and assisting the turnaround of the lowest-performing 
schools in the state, meaning that in many cases, either the introduction of the ISD initiative is 
not a new concept or the impacts of previous initiatives in potential comparison schools still 
linger. As a result, while we continue to take great care in our selection of comparison schools, 
we also will continue to present all conclusions from our analyses with a strong word of 
caution. 

Propensity Score Matching 

Linden (2015), Rubin (2 001), 33 and others recommend using a statistical process known as 
propensity score matching (PSM) for identifying members of comparison groups for analyses 
like the ones we proposed to use (Appendix F). Many researchers suggest that the specific PSM 


32 Somers, M., Zhu, P., Jacob, R., and Bloom, H. (2013). The Validity and Precision of the Comparative Interrupted 
Time Series Design and the Difference-in-Difference Design in Educational Evaluation. Working Paper. New York 
and Oakland, CA: MDRC. 

33 Linden, A. (2015). Conducting Interrupted Time-Series Analysis for Single- and Multiple-Group Comparisons. The 
Stata Journal, 15(2): 480-500; Rubin, D. B. (2001). Using Propensity Scores to Help Design Observational Studies: 
Application to the Tobacco Litigation. Health Services and Outcomes Research Methodology, 2(3-4): 169- 

188. 
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strategy—and there are several —matters much less than does the choice of variables on which 
schools are matched (see, for instance, Hallberg et al. 2018 34 ). In addition to including pre¬ 
intervention measures of the outcomes of interest as part of the matching process (in our case, 
student testing outcomes), our matching also includes demographic covariates that change 
over time or are likely to have been impacted by historical changes outside the scope of the 
initiative, to reduce the influence of those factors on analyses of the outcomes of interest 
(Hallberg et al. 2018). 

Based on the findings of Somers et al. (2013), since we have a large candidate pool of schools 
relative to the treated schools, and since we have more than two years of pre-intervention test 
data, we use a radius matching (propensity scores within 0.25 SD of each treatment school's 
score) with replacement, which matches each treatment school to several schools within a 
given propensity score range, increases the size of the comparison pool, and likely has little 
impact on bias because of the depth of pre-intervention data available for matching. 


34 Hallberg, K., Williams, R., Swanlund, A., and Eno, J. (2018). Short Comparative Interrupted Time Series Using 
Aggregate School-Level Data in Education Research. Educational Researcher, 47(5): 295-306. 
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Appendix H: Survey Instruments 


8/15/2019 


Qualtrics Survey Software 


Intro 


ISD Evaluation 
Teacher & Staff Survey 

Thank you for your participation in the North Carolina Innovation School District 
evaluation survey. Our questions are intended to give us more information about the 
impact of the ISD program, which your school is part of this year. 

There are no right or wrong answers; we encourage your honest and candid responses. 
All responses are kept strictly confidential. In reports, all responses will be combined, so 
no one will be able to connect you to your responses. 

Your participation is entirely voluntary and you may exit the survey at any time. We 
appreciate your willingness to participate and thank you in advance for your insight. 

If you have questions or technical difficulty while completing the survey, please contact 
Trip Stallings at the Friday Institute, NC State University, by telephone (919.513.8576) or 
by email f dtstalli @ ncsu.edu) . 


Consent 


Click here for a downloadable copy of this consent from. 


North Carolina State University 
INFORMED CONSENT FORM for RESEARCH 

Educator Consent Form 

Title of Study: North Carolina Innovation School District Evaluation 
Principal Investigator: Dr. Trip Stallings 


What is the purpose of this study? 

On behalf of The North Carolina Department of Public Instruction (NCDPI), the Friday Institute at North Carolina State University is evaluating 
the Innovative School District (ISD). The ISD operates as a separate, non-geographical school district within NCDPI, with the purpose of 
improving chronically low-performing schools throughout the state. The ISD’s charge is to work with identified schools and their communities to 
foster accountable, data-driven partnerships designed to promote and implement a shared vision of equity and opportunity for students in those 
schools. The intent of the evaluation is to help school, ISD, and NCDPI leadership understand the impact of the ISD. 

https://ncsu.ca1.quaitrics.eom/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 1/6 
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8/15/2019 


Qualtrics Survey Software 


What will happen if you take part in the study? 

Teachers and Administrators will be invited to participate in surveys and focus group interviews to discuss their impressions of and experiences 
with the pilot program. Surveys will take approximately 10-20 minutes to complete; group interviews will take approximately 30-60 minutes to 
complete. Participants will be asked to convene at a time that is least disruptive to their day-to-day work responsibilities. All interviews will be 
coordinated with local school staff and conducted on school grounds. 


Risks 

There are no big risks in taking part in this study. 

Benefits 

There are no direct benefits, but the results from this study will help determine the impact of the ISD program and help inform future education 
models. 

Will there be any audio/video recording? 

The interviews will be digitally audio- recorded and then transcribed. After transcription, the recordings will be destroyed. 

Confidentiality 

All information collected for this study will be stored in secure, password-protected locations at NC State University, and only researchers at NC 
State will be able to see it. The researchers will never share information from the study that will allow people to connect what is shared with the 
individual who shared it. 

Compensation 

None. 

What if you have questions about this study? 

If you have questions at any time about the study, you may contact the research coordinator. Trip Stallings (919.513.8576, dtstalli@ncsu.edu) at the 
Friday Institute for Educational Innovation, North Carolina State University, 1890 Main Campus Road, Campus Box 7249, Raleigh, NC 27606. 


Consent to Participate 

"I have read and understand the above information. I have received a copy of this form. I 
agree to participate in this study with the understanding that I may choose not to 
participate or to stop participating at any time without penalty or loss of benefits to which 
I am otherwise entitled." 

O Yes, I agree to participate in the study with the understanding that I may withdraw at any time, 
o No, I decline to participate. 


Demographics 


What is the name of the school where you work this year? 
o Southside-Ashpole Elementary School 


What grade(s) do you teach this year? 


https://ncsu.ca1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 
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8/15/2019 Qualtrics Survey Software 

|—| Kindergarten 

□ 1 
□ 2 

□ 3 

□ 4 

□ 5 

l~l Other (please specify): 

I identify as: 

O Female 
O Male 
O Other 

0 I decline to answer 


I identify as: 

0 American Indian or Alaska Native (Not Hispanic or Latino) - A person having origins in any of the 
original peoples of North and South America (including Central America), and who maintains tribal 
affiliation or community attachment. 

O Asian (Not Hispanic or Latino) - A person having origins in any of the original peoples of the Far 
East, Southeast Asia, or the Indian Subcontinent, including for example, Cambodia, China, India, 
Japan, Korea, Malaysia, Pakistan, the Philippine Islands, Thailand, or Vietnam. 

o Black or African American (Not Hispanic or Latino) - A person having origins in any of the black 
racial groups of Africa. 

0 Hispanic or Latino - A person having ethnic origins in Latin America and the Iberian Peninsula, 
including the Carribbean. 

o Native Hawaiian or Other Pacific Islander (Not Hispanic or Latino) - A person having origins in 
any of the peoples of Hawaii, Guam, Samoa, or other Pacific Islands. 

o White (Not Hispanic or Latino) - A person having origins in any of the original peoples of Europe, 
the Middle East, or North Africa. 

O Two or more races/ethnicities - A person who identifies with two or more of the above. 

O I decline to answer 


Survey Questions 

https://n csu.cal.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 
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Qualtrics Survey Software 


Instructions 


• For each statement, select the box response best matches how much you 
agree with the statement. 

• If you do not have enough information to form an opinion on an item, or if an 
item does not apply to your situation, select "Do Not Know or Not 
Applicable." 

• If you have enough information to form an opinion but are split between 
"Agree" and "Disagree," select "Neutral." 


At ${q://QID6/ChoiceGroup/SelectedChoices} this year... 


Strongly Strongly 

Agree Agree Neutral Disagree Disagree 


1 have opportunities to participate in grade-level and 
school-level planning committees. 

o 

o 

o 

o 

o 

School leadership encourages teachers and staff to 
take leadership opportunities. 

o 

o 

o 

o 

o 

School leadership asks me for their opinions. 

o 

o 

o 

o 

o 

If 1 have a concern about a curriculum, policy, or 
other decision, 1 feel comfortable talking to someone 
in school leadership. 

o 

o 

o 

o 

o 

1 have a say in choosing the curriculum 1 use in my 
classroom. 

o 

o 

o 

o 

o 

All staff members are treated equitably. 

o 

o 

o 

o 

o 


At ${q://QID6/ChoiceGroup/SelectedChoices} this year... 


https://n csu.cal.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 
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Qualtrics Survey Software 


§tF8R§ty §tF8R|ly 

Aifii A§f§i Neutral gisigFii §i§i§F§§ 


1 feel connected to the community my school 
serves. 

o 

o 

o 

o 

o 

1 communicate with parents/guardians on a regular 
basis. 

o 

o 

o 

o 

o 

1 have parent/guardian volunteers in my classroom. 

o 

o 

o 

o 

o 


At ${q://QID6/ChoiceGroup/SelectedChoices} this year... 


I use student data to improve instruction. 

There is a process in place to help all staff 
members improve. 

The professional development offered by my school 
is high-quality. 


The professional development offered by my school 
is relevant to my needs. 

I have enough time in my schedule each week for 
lesson planning. 

I have time set in my schedule to plan and 
collaborate with other teachers and staff. 


Strongly Strongly 


Agree 

Agree 

Neutral 

Disagree 

Disagree 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 


o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 


https://n csu.cal.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 
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Qualtrics Survey Software 


Intro 


English 


▼ 


ISD Evaluation 
Parent/Guardian Survey 

Thank you for taking our survey about your child's/children's school. This survey is part 
of an evaluation of the North Carolina Innovation School District, which Southside- 
Ashpole Elementary School is part of this year. Our questions are intended to give us 
more information about the impact of the ISD program. 

There are no right or wrong answers; we encourage your honest and candid responses. 
All responses are kept strictly confidential. In reports, all responses will be combined, so 
no one will be able to connect you to your responses. 

Your participation is entirely voluntary and you may exit the survey at any time. We 
appreciate your willingness to participate and thank you in advance for your insight. 

If you have questions or technical difficulty while completing the survey, please contact 
Trip Stallings at the Friday Institute, NC State University, by telephone (919.513.8576) or 
by email (dtstalli@ncsu.edu). 


Consent 


Click here for a downloadable copy of the consent form in English and Spanish. 


North Carolina State University 
INFORMED CONSENT FORM for RESEARCH 

Parent Consent Form 

Title of Study: North Carolina Innovation School District Evaluation 
Principal Investigator: Dr. Trip Stallings 

https://ncsu.ca1.qualtrics.eom/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 1/7 
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8/15/2019 Qualtrics Survey Software 

What is the purpose of this study? 

On behalf of The North Carolina Department of Public Instruction (NCDPI), the Friday Institute at North Carolina State University is evaluating 
the Innovative School District (ISD). The ISD operates as a separate, non-geographical school district within NCDPI, with the purpose of 
improving chronically low-performing schools throughout the state. The ISD’s charge is to work with identified schools and their communities to 
foster accountable, data-driven partnerships designed to promote and implement a shared vision of equity and opportunity for students in those 
schools. The intent of the evaluation is to help school, ISD, and NCDPI leadership understand the impact of the ISD. 

What will happen if you take part in the study? 

Parents and guardians will be invited to participate in surveys to discuss their experiences with the ISD. The surveys will take approximately 10 
minutes to complete. 

Risks 

There are no big risks in taking part in this study. 

Benefits 

There are no direct benefits, but the results from this study will help determine the impact of the ISD program and help inform future education 
models. 

Confidentiality 

All information collected for this study will be stored in secure, password-protected locations at NC State University, and only researchers at NC 
State will be able to see it. The researchers will never share information from the study that will allow people to connect what is shared with the 
individual who shared it. 

Compensation 

None. 

What if you have questions about this study? 

If you have questions at any time about the study, you may contact the research coordinator. Trip Stallings (919.513.8576, dtstalli@ncsu. edu) at the 
Friday Institute for Educational Innovation, North Carolina State University, 1890 Main Campus Road, Campus Box 7249, Raleigh, NC 27606. 


Consent to Participate 

"I have read and understand the above information. I have received a copy of this form. I 
agree to participate in this study with the understanding that I may choose not to 
participate or to stop participating at any time without penalty or loss of benefits to which 
I am otherwise entitled." 

0 Yes, I agree to participate in the study with the understanding that I may withdraw at any time, 
o No, I decline to participate. 


Demographics 

Select the name of the school that at least one of your children attends from the list 
below. 

o Southside-Ashpole Elementary School 
O My child does not attend any of these schools 


https://n csu.cal.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 
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What grade(s) is your child/are your children in this year 
at ${q://QID6/ChoiceGroup/SelectedChoices}? If you have more than one child 
at ${q://QI D6/ChoiceGroup/SelectedChoices}, select all of the grades that apply. 

□ Kindergarten 

□ 1 
□ 2 

□ 3 

□ 4 

□ 5 


I identify as: 

o American Indian or Alaska Native (Not Hispanic or Latino) - A person having origins in any of the 
original peoples of North and South America (including Central America), and who maintains tribal 
affiliation or community attachment. 

O Asian (Not Hispanic or Latino) - A person having origins in any of the original peoples of the Far 
East, Southeast Asia, or the Indian Subcontinent, including for example, Cambodia, China, India, 
Japan, Korea, Malaysia, Pakistan, the Philippine Islands, Thailand, or Vietnam. 

o Black or African American (Not Hispanic or Latino) - A person having origins in any of the black 
racial groups of Africa. 

o Hispanic or Latino - A person having ethnic origins in Latin America and the Iberian Peninsula, 
including the Carribbean. 

o Native Hawaiian or Other Pacific Islander (Not Hispanic or Latino) - A person having origins in 
any of the peoples of Hawaii, Guam, Samoa, or other Pacific Islands. 

0 White (Not Hispanic or Latino) - A person having origins in any of the original peoples of Europe, 
the Middle East, or North Africa. 

o Two or more races/ethnicities - A person who identifies with two or more of the above. 

O I decline to answer 


Survey Questions: Likert 


Instructions 


https://n csu.cal.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 
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• For each statement, select the response that best matches how much you 
agree with the statement. 

• If you do not have enough information to form an opinion on an item, or if an 
item does not apply to your situation, select "Do Not Know or Not 
Applicable." 

• If you have enough information to form an opinion but are split between 
"Agree" and "Disagree," select "Neutral." 

• If you have more than one child at ${q://QID6/ChoiceGroup/SelectedChoices} 
this year, select the response that best matches how you feel in general about 
your experience with the school this year. 


At ${q://QID6/ChoiceGroup/SelectedChoices} this year... 


Strongly Strongly 



Agree 

Agree 

Neutral 

Disagree 

Disagree 

My child is safe at school. 

o 

o 

o 

0 

0 

If my child feels threatened at school, adults at the 
school will help him/her. 

o 

o 

o 

0 

0 

If my child is bullied outside of school (for instance, 
online), he/she can get help from an adult at school. 

o 

0 

o 

0 

0 


With regard to ${q://QID6/ChoiceGroup/SelectedChoices} this year... 


If I tell my child’s teachers about a concern with my 
child (for instance, related to academic challenges, 
behavior issues, etc.), they will try to address my 
concern. 

If I tell the school administrators about a concern 
with my child (for instance, related to academic 
challenges, behavior issues, etc.) or the school, 
they will try to address my concern. 

I feel comfortable talking to my child’s teachers. 

https://n csu.cal.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 


Strongly Strongly 

Agree Agree Neutral Disagree Disagree 

o o o o o 

o o o o o 

o o o o o 
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I feel comfortable talking to administrators at my 
child’s school. 

I know what to do when problems arise with my 
child in school. 

I feel welcomed at the school. 


Strongly 

Agree 

Agree 

Neutral 

Disagree 

Strongly 

Disagree 

0 

0 

o 

0 

0 

0 

0 

o 

o 

0 

o 

o 

o 

o 

o 


With regard to ${q://QID6/ChoiceGroup/SelectedChoices} this year... 


The school provides me with clear information 
about what my child is learning in school. 

The school provides me with clear information 
about how to support my child's learning at home. 

I have a good idea of how my child is doing in 
school. 

I regularly talk to my child about what he/she is 
learning in school. 

I assist my child with his/her homework. 

The school has helped me learn new ways to help 
my child grow and develop in school. 


Strongly 

Agree 

Agree 

Neutral 

Disagree 

Strongly 

Disagree 

0 

0 

o 

o 

0 

0 

0 

o 

0 

0 

0 

0 

o 

0 

0 

0 

0 

o 

o 

0 

o 

o 

o 

o 

o 

0 

0 

o 

0 

0 


The primary language we use at home is: 

O English 
O Spanish 
O Other 


Wth regard to ${q://QID6/ChoiceGroup/SelectedChoices} this year... 


https://n csu.cal.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 


5/7 


The William and Ida Friday Institute for Educational Innovation 


68 




Innovative School District: Year 1 Evaluation 


DRAFT - September 2019 


8/15/2019 


Qualtrics Survey Software 



Strongly 

Agree 

Agree 

Neutral 

Disagree 

Strongly 
Disagree P 

The school communicates with me in the 
language my family uses at home. 

0 

0 

o 

0 

0 


Survey Questions: Frequency 


Instructions 

• For each statement, select the box response comes closest to describing how 
often you do each activity. 

• If you do not have enough information to form an opinion on an item, select 
"Do Not Know." 

• If you have more than one child at ${q://QID6/ChoiceGroup/SelectedChoices} 
this year, select the response that best matches how you feel in general about 
your experience with the school this year. 


As a parent of a child/children at ${q://QID6/ChoiceGroup/SelectedChoices}, during this 
school year, how often have you done each of the following activities? 



Never 

Once 

or 

Twice 

Once 

Every 

Few 

Months 

Monthly 

Weekly 
or More 

Meet in person with teachers at your child's 
school 

o 

o 

o 

o 

o 

Visit your child's school 

o 

0 

o 

o 

0 

Participate in decision-making groups, 
committees, or advisory councils at your child's 

o 

0 

o 

o 

0 


school 


Survey Questions: Importance 


https://ncsu.ca1.qualtrics.com/Q/EditSection/Blocks/Ajax/GetSurveyPrintPreview 6/7 
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Instructions 


• For each statement, select the box that comes closest to describing how 
important you feel each activity is. 

• If you do not have enough information to form an opinion on an item, select 
"Do Not Know." 


How important is each of these to your child's/children's success in school? 



Absolutely 

Essential 

Very 

Important 

Of Average 
Importance 

Slightly 

Important 

Not 

Important 

At All 

Do Not 
Know 

Attending school 
regularly 

o 

o 

o 

o 

o 

o 

Completing homework 
regularly 

o 

o 

0 

o 

o 

0 


Powered by Qualtrics 
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ISD Evaluation 
Student Survey 

Thank you for taking our survey about your school. 

Please answer the survey honestly and truthfully. There are no right or wrong 
answers. You will not be asked for your name so no one will know what you said. 


Your participation is voluntary and you may stop taking the survey at any time. We 
appreciate your participation and thank you in advance for your help. 


If you have questions while completing the survey, please contact Trip Stallings by 
telephone (919.513.8576) or by email (dtstalli@ncsu.edu). 

Circle the school and grade you are in this year . 

A. What school do you go to? 

Southside-Ashpole Elementary School 

B. What grade are you in? 

Grade 3 Grade 4 Grade 5 
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Instructions 

• Circle the answer that best matches how much you agree with each statement. 

• If you are not sure how you feel about a statement, circle “I Don’t Know.” 

• If you agree with the statement some of the time and disagree some of the time, circle 

“Sometimes Agree/Sometimes Disagree.” 


1. I feel safe at my school. 

Agree Sometimes Agree/Sometimes Disagree Disagree I Don’t Know 


2. If I feel threatened at school, adults at my school will help me. 

Agree Sometimes Agree/Sometimes Disagree Disagree I Don’t Know 


3. If I am bullied outside of school by someone from my school (in person or online), adults at 
my school will help me. 

Agree Sometimes Agree/Sometimes Disagree Disagree I Don’t Know 


4. Overall, adults at my school treat students fairly. 

Agree Sometimes Agree/Sometimes Disagree Disagree I Don’t Know 


5. Adults in my school treat me with respect. 

Agree Sometimes Agree/Sometimes Disagree Disagree I Don’t Know 


6. Adults at my school listen to students. 

Agree Sometimes Agree/Sometimes Disagree Disagree I Don’t Know 
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7. At my school, teachers care about students. 

Agree Sometimes Agree/Sometimes Disagree Disagree 


8. Adults at my school are available if I need help. 

Agree Sometimes Agree/Sometimes Disagree Disagree 


9. In general, I enjoy going to school. 

Agree Sometimes Agree/Sometimes Disagree Disagree 


10. Most teachers take time to answer my questions. 

Agree Sometimes Agree/Sometimes Disagree Disagree 


11. Most teachers grade my assignments and tests fairly. 

Agree Sometimes Agree/Sometimes Disagree Disagree 


I Don’t Know 


I Don’t Know 


I Don’t Know 


I Don’t Know 


I Don’t Know 
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Appendix I: Survey Results 


Survey Response Key. 

SD: Strongly Disagree 

D: Disagree 

N: Neutral 

SA/SD: Sometimes Agree, Sometimes Disagree 

A: Agree 

SA: Strongly Agree 

IDK: I Don't Know 


Parent/Guardian and Student Common Questions 

Q1 My child is safe at school. / I feel safe at my school. 



SD 

D 

N 

A 

SA 

IDK 

SD + D 

N 

A+SA 

n 

Parent/Guardian 

6% 

18% 

12% 

47% 

18% 

0% 

24% 

12% 

65% 

17 

Student* 

- 

11% 

22% 

59% 

- 

9% 

11% 

22% 

59% 

46 

Q2 

If my child feels threatened at school, adults at the school will help him/her. / If 1 feel threatened at school, adults at my school will help me. 
SD D N A SA IDK SD + D N A + SA 

n 

Parent/Guardian 

12% 

12% 

24% 

35% 

18% 

0% 

24% 

24% 

53% 

17 

Student* 

- 

13% 

9% 

67% 

- 

11% 

13% 

9% 

67% 

46 

Q3 

If my child is bullied outside of school (for instance, online), he/she can get help from an adult at school. / 

If 1 am bullied outside of school (for instance, online) by someone from my school, adults at my school will help me. 

SD D N A SA IDK SD + D N 

A+SA 

n 

Parent/Guardian 

18% 

0% 

18% 

35% 

24% 

6% 

18% 

18% 

59% 

17 

Student* 

- 

17% 

15% 

54% 

- 

13% 

17% 

15% 

54% 

46 

*The student survey uses a limited response set: 

Agree, Sometimes Agree/Sometimes Disagree, Disagree, and 1 Don't Know. Here, Sometimes Agree/Sometimes Disagree is recorded as Neutral. 


The William and Ida Friday Institute for Educational Innovation 


74 





















Innovative School District: Year 1 Evaluation 


DRAFT - September 2019 


Parent 

Qi 


Q2 


Q3 


Q4 


Q5 


Q6 


Q7 


Q8 


Q9 


My child is safe at school. 
SD D 


N 


SA 


IDK 


6 % 


18% 


12 % 


47% 


18% 


0 % 


If my child feels threatened at school, adults at the school will help him/her. 
SD D N A SA IDK 


12 % 


12 % 


24% 


35% 


18% 


0 % 


If my child is bullied outside of school (for instance, online), he/she can get help from an adult at school. 
SD D N A SA IDK 


18% 


0 % 


18% 


35% 


24% 


6 % 


SD 


N 


SA 


IDK 


6 % 


0 % 


29% 


29% 


35% 


0 % 


SD + D 

N 

A+SA 

n 

24% 

12% 

65% 

17 

SD + D 

N 

A+SA 

n 

24% 

24% 

53% 

17 

SD + D 

N 

A+SA 

n 

18% 

18% 

59% 

17 

avior issues, etc.), they will try to address my concern. 
SD + D N A+SA n 

6% 

29% 

64% 

17 


If I tell the school administrators about a concern with my child (for instance, related to academic challenges, behavior issues, etc.) or the school, they will try to address my concern. 


SD 


D 


N 


SA 


6 % 


6 % 


29% 


35% 


24% 


I feel comfortable talking to my child's teachers. 
SD D N A 


SA 


0 % 


6 % 


18% 


35% 


41% 


0 % 


12 % 


18% 


47% 


24% 


0 % 


6 % 


6 % 


41% 


47% 


I feel welcomed at the school. 

SD D N 


SA 


0 % 


0 % 


18% 


47% 


35% 


IDK 


0 % 


IDK 


0 % 


I feel comfortable talking to administrators at my child's school. 

SD D N A SA IDK 


0 % 


I know what to do when problems arise with my child in school. 

SD D N A SA IDK 


0 % 


IDK 


0 % 


SD + D 

N 

A+SA 

n 

12% 

29% 

59% 

17 

SD + D 

N 

A+SA 

n 

6% 

18% 

76% 

17 

SD + D 

N 

A+SA 

n 

12% 

18% 

71% 

17 

SD + D 

N 

A+SA 

n 

6% 

6% 

88% 

17 

SD + D 

N 

A+SA 

n 

0% 

18% 

82% 

17 
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Q10 The school provides me with clear information about what my child is learning school. 


Qll 


Q12 


Q13 


Q14 


Q15 


Q16 


Q17 


SD_D_ N _ A _ SA _ IDK_ 

6% 0% 41% 24% 29% 0% 


The school provides me with clear information about how to support my child's learning at home. 

SD _D_ N _ A _ SA _ IDK 

6% 0% 47% 18% 29% 0% 


I have a good idea of how my child is doing in school. 

SD _D_ N _ A _ SA _/D/C 

6% 0% 29% 41% 24% 0% 


I regularly talk to my child about what he/she is learning in school. 

SD _D_ N _ A _ SA _ IDK_ 

0% 6% 0% 47% 47% 0% 


I assist my child with his/her 
homework. 

SD _D_ N _ A _ SA _ IDK_ 

0% 0% 12% 41% 35% 12% 


The school has helped me learn new ways to help my child grow and develop in school. 

SD _D_ N _ A _ SA _ IDK 

6% 6% 29% 29% 24% 6% 


The primary language we use at home is: 
English Spanish Other 

100 % 0 % 0 % 


The school communicates with me in the language my family uses at home. 

SD _D_ N _ A _ SA _/D/C 

0% 0% 6% 53% 41% 0% 


SD + D 

N 

A+SA 

n 

6% 

41% 

53% 

17 


SD + D 

N 

A+SA 

n 

6% 

47% 

47% 

17 


SD + D 

N 

A+SA 

n 

6% 

29% 

65% 

17 


SD + D 

N 

A+SA 

n 

6% 

0% 

94% 

17 


SD + D 

N 

A+SA 

n 

0% 

12% 

76% 

17 


SD + D 

N 

A+SA 

n 

12% 

29% 

53% 

17 


n 




17 




SD + D 

N 

A+SA 

n 

0% 

6% 

94% 

17 


As a parent of a child/children at Southside Ashpole Elementary School, during this school year, how often have you done each of the following activities? 


Q18 Meet in person with teachers at your child's school. 

Once Every 

Few Weekly or 

Never Once/Twice Months Monthly More 


n 


0% 47% 18% 24% 12% 


17 


The William and Ida Friday Institute for Educational Innovation 


76 



Innovative School District: Year 1 Evaluation 


DRAFT - September 2019 


Q19 


Q20 


Visit you child's school. 


Once Every 


Never 

Once/Twice 

Few 

Months 

Monthly 

Weekly or 

More 

n 

0% 

12% 

24% 

24% 

41% 

17 

Participate in decision-making groups, committees, or advisory councils at your child's school. 




Once Every 






Few 


Weekly or 


Never 

Once/Twice 

Months 

Monthly 

More 

n 

11% 

12% 

6% 

6% 

0% 

17 


How important is each of these to your child's/children's success in school? 


Q21 


Q22 


Attending school regularly. 


Not 


Absolutely 

Very 

Of Average 

Slightly 

Important 

Do Not 

Essential 

Important 

Importance 

Important 

At All 

Know 

11% 

24% 

0% 

0% 

0% 

6% 


Completing homework regularly. 




Absolutely 

Essential 

Very 

Important 

Of Average 
Importance 

Slightly 

Important 

Not 

Important 

At All 

Do Not 

Know 

11% 

24% 

0% 

0% 

0% 

6% 


Student 

Q1 I feel safe at my school. 

D _ SA/SD _ A _ IDK_ 

11% 22% 59% 9% 


Q2 If I feel threatened at school, adults at my school will help me. 

D _ SA/SD _ A _ IDK 

13% 9% 67% 11% 


n 

46 


n 

46 


Q3 If I am bullied outside of school by someone from my school (in person or online, adults at my school will help me. 

D _ SA/SD _ A _ IDK n 

17% 15% 54% 13% 46 


The William and Ida Friday Institute for Educational Innovation 


77 



Innovative School District: Year 1 Evaluation 


DRAFT - September 2019 


Q4 Overall, adults at my school treat students fairly. 

D _ SA/SD _ A _ IDK _ n_ 

16% 13% 64% 7% 45 


Q5 Adults in my school treat me with respect. 

D _ SA/SD _ A _ IDK _ n_ 

13% 11% 67% 9% 46 


Q6 Adults at my school listen to students. 

D _ SA/SD _ A _ IDK _n_ 

13% 28% 57% 2% 46 


Q7 At my school, teachers care about students. 

D _ SA/SD _ A _ IDK _ n_ 

7% 1% 18% 9% 45 


Q8 Adults at my school are available if I need help. 

D _ SA/SD _ A _ IDK _ n_ 

9% 11% 72% 9% 46 


Q9 In general, I enjoy going to school. 

D _ SA/SD _ A _ IDK_ 

24% 33% 39% 4% 


Q10 Most teachers take time to answer my questions. 

D _ SA/SD _ A _ IDK_ 

24% 22% 48% 7% 


Qll Most teachers grade my assignments and tests fairly. 

D _ SA/SD _ A _ IDK 

7% 9% 78% 7% 


Q12 What grade are you in this year? 

3 _ 4 _5 

33% 33% 35% 


n 

46 


n 

46 


n 

46 


n 

46 
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Teachers 

Qi 


Q2 


Q3 


Q4 


Q5 


Q6 


Q7 


Q8 


Q9 


I have opportunities to participate in grade level and school-level planning committees. 

SD _D_ N _ A _ SA _ IDK SD + D _ N _ A + SA _ n_ 

0% 8% 8% 42% 42% 0% 8% 8% 83% 12 


School leadership encourages teachers and staff to take leadership opportunities. 

SD _D_ N _ A _ SA _ IDK SD + D _ N _ A + SA _ n_ 

0% 8% 8% 58% 25% 0% 8% 8% 83% 12 


School leadership asks me for their opinions. 

SD _D_ N _ A _ SA _ IDK SD + D _ N _ A + SA _ n_ 

8% 17% 8% 3% 25% 8% 25% 8% 28% 12 


If I have a concern about a curriculum, policy, or other decision, I feel comfortable talking to someone in school leadership. 

SD _D_ N _ A _ SA _ IDK SD + D _ N _ A + SA _n_ 

8% 8% 17% 25% 42% 0% 17% 17% 67% 12 


I have a say in choosing the curriculum I use in my classroom. 

SD _D_ N _ A _ SA _ IDK SD + D _ N _ A + SA _ n_ 

9% 18% 18% 36% 9% 9% 27% 18% 45% 11 


All staff members are treated equitably. 

SD _D_ N _ A _ SA _ IDK SD + D _ N _ A + SA _ n_ 

0% 27% 9% 27% 27% 9% 27% 9% 55% 11 


I feel connected to the community my school serves. 

SD _D_ N _ A _ SA _ IDK SD + D _ N _ A + SA _ n_ 

0% 8% 25% 42% 17% 8% 8% 25% 58% 12 


I communicate with parents/guardians on a regular basis. 

SD _D_ N _ A _ SA _ IDK SD + D _ N _ A + SA _ n_ 

0% 0% 0% 58% 42% 0% 0% 0% 100% 12 


I have parent/guardian volunteers in my classroom. 

SD _D_ N _ A _ SA _ IDK SD + D _ N _ A + SA _ n_ 

0% 42% 17% 17% 17% 8% 42% 17% 33% 12 
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Q10 


Qll 


Q12 


Q13 


Q14 


Q15 


I use student data to improve instruction. 

SD _ D _ N _ A _ SA _/D/C 

0% 0% 0% 42% 58% 0% 


There is a process in place to help all staff members improve. 

SD _D_ N _ A _ SA _/D/C 

0% 0% 42% 50% 8% 0% 


The professional development offered by my school is high-quality. 

SD _D_ N _ A _ SA _/D/C 

0% 8% 25% 58% 8% 0% 


The professional development offered by my school is relevant to my needs. 

SD _D_ N _ A _ SA _/D/C 

0% 8% 33% 42% 17% 0% 


I have enough time in my schedule each week for lesson planning. 

SD _D_ N _ A _ SA _/D/C 

0% 25% 33% 25% 17% 0% 


I have time set in my schedule to plan and collaborate with other teachers and staff. 

SD _D_ N _ A _ SA _ IDK 

0% 8% 25% 33% 33% 0% 


SD + D 

N 

A+SA 

n 

0% 

0% 

100% 

12 


SD + D 

N 

A+SA 

n 

0% 

42% 

58% 

12 


SD + D 

N 

A+SA 

n 

8% 

25% 

67% 

12 


SD + D 

N 

A+SA 

n 

8% 

33% 

58% 

12 


SD + D 

N 

A+SA 

n 

25% 

33% 

42% 

12 


SD + D 

N 

A+SA 

n 

8% 

25% 

67% 

12 


The William and Ida Friday Institute for Educational Innovation 


80 



Innovative School District: Year 1 Evaluation 


DRAFT - September 2019 


Appendix J: Recommended School-Level Annual Report Components 

Q2. Does the ISD improve learning conditions, including changes in student behavior? 

• Description of Exceptional Children (EC) plan (outlining the full continuum of services 
to meet the special education and related services needs of students with disabilities) 

• Proof of a highly-qualified and licensed special education teacher(s) and/or director 

• Identification of sources for external EC service providers (e.g., school psychologist, 
occupational therapist, physical therapist, speech language pathologist, etc.) 

• Evidence of on-site/accessible, appropriate instructional supports for students with 
disabilities (as indicated in lEPs) 

• Statement of school policies for EC, incorporating all required elements to meet Federal 
and State regulations (e.g., discipline, confidentiality, accountability, maintenance of 
effort, security and confidentiality of EC Student Special Education files, etc.) 

• Address transportation (as a related service) for EC students who may have this as a 
component of their IEP 

• Description of school's approach to discipline and changes over time 

• Description of school's responses to attendance problems/truancy 

Q3. Does the ISD contribute to changes in school-community engagement? 

• Description of the process of identifying valuable external partners 

• Annual list of partners and description of services they provide, including level of 
partnerships (time commitment, financial commitment) 

• If evaluation budget supports: Descriptive report of offsetting services or additional 
value in services provided; where available, budget data on any financial offsets 
arising from partnerships 

• Annual list of parent/guardian involvement in school events 

Q4a. Does the ISD contribute to an overall change in the culture of schooling in ISD schools? 

• Plan for diffusion of best practices 

Q4b. Does the ISD change approaches to ensuring sustainable, high quality staffing in ISD schools? 

• Description of teacher recruitment and retention processes 

• ISO protocol for teacher evaluation 

• Annual list of professional development opportunities (plus attendance figures) 

Q4c. Does the ISD change approaches to leadership in ISD schools? 

• Annual school operating costs, compared to average operating costs for similarly- 
sized new schools and/or new charter schools 

• Description of teacher leadership model 

• Evidence of teacher leadership model implementation 

Q4d. Does the ISD contribute to changes in the culture of schooling in partnering LEAs, non¬ 
participating LEAs, and/or across the State overall ? 

• Plan for diffusion of best practices to partner LEAs 

Key : Red text=indicator for legislatively-highlighted outcomes 
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